2022-09-13
§
|
16:30 |
<btullis> |
restarting hive-server2 and hive-metastore on an-coord1001 (currently standby) |
[analytics] |
16:29 |
<btullis> |
restarting oozie on an-coord1001 |
[analytics] |
16:10 |
<joal> |
Rerun failed oozie webrequest jobs |
[analytics] |
15:57 |
<btullis> |
rolling out updated hadoop packages to an-airflow1003 |
[analytics] |
15:55 |
<btullis> |
rolling out upgraded hadoop client packages to stat servers. |
[analytics] |
15:51 |
<btullis> |
restarting eventlogging_to_druid_network_flows_internal_hourly.service eventlogging_to_druid_prefupdate_hourly.service refine_event_sanitized_analytics_immediate.service refine_event_sanitized_main_immediate.service |
[analytics] |
15:49 |
<btullis> |
restarting eventlogging_to_druid_navigationtiming_hourly.service on an-launcher1002 |
[analytics] |
15:46 |
<btullis> |
restarting eventlogging_to_druid_editattemptstep_hourly.service on an-launcher1002 |
[analytics] |
15:44 |
<btullis> |
cancel that last message. Upgrading hadoop packages on an-launcher instead. They were inadvertently omitted last time. |
[analytics] |
15:39 |
<btullis> |
Going to downgrade hadoop on ann hadoop-worker nodes to 2.10.1 |
[analytics] |
15:21 |
<btullis> |
failed over hive to an-coord1002 via DNS https://gerrit.wikimedia.org/r/c/operations/dns/+/831906 |
[analytics] |
15:20 |
<btullis> |
restarted yarn service on an-master1002 to make the active host an-master1001 again. |
[analytics] |
15:11 |
<btullis> |
restart hive-server2 and hive-metastore service on an-coord1002 to pick up new version of hadoop |
[analytics] |
14:55 |
<btullis> |
rolling out updated hadoop packages to analytics-airflow (cumin alias) hosts |
[analytics] |
14:42 |
<btullis> |
sudo systemctl restart analytics-reportupdater-logs-rsync.service on an-launcher1002 |
[analytics] |
13:21 |
<joal> |
Manual launch of refinery-drop-mediawiki-snapshots with new tables in patch https://gerrit.wikimedia.org/r/831866 |
[analytics] |
10:51 |
<btullis> |
attempting failback operation on hadoop namenodes |
[analytics] |
09:42 |
<btullis> |
roll-restarting the hadoop masters via the cookbook |
[analytics] |