751-800 of 6097 results (33ms)
2023-07-05 §
11:45 <btullis> restarted hive-servers2 and hive-metastore service on an-coord1002 [analytics]
11:40 <btullis> roll-restarting kafka-jumbo brokers for T329716 [analytics]
11:01 <btullis> roll-restarting the presto workers for T329716 [analytics]
10:20 <btullis> deploying updated spark3 defaults to disable the `spark.shuffle.useOldFetchProtocol`option for T332765 [analytics]
09:45 <btullis> failing back namenode to an-master1001 with `sudo -u hdfs /usr/bin/hdfs haadmin -failover an-master1002-eqiad-wmnet an-master1001-eqiad-wmnet` on an-master1001 [analytics]
09:38 <btullis> re-enabled gobblin jobs on an-launcher1002 [analytics]
09:03 <btullis> switching yarn shuffler - running puppet on 87 worker nodes [analytics]
08:44 <btullis> disabled gobblin and spark jobs on an-launcher for T332765 [analytics]
08:33 <btullis> disabled gobblin jobs with https://gerrit.wikimedia.org/r/c/operations/puppet/+/935425 [analytics]
08:27 <btullis> roll-restarting hadoop workers in the test cluster [analytics]
2023-07-04 §
13:55 <btullis> roll-restarting the eventgate-analytics-external worker pods in eqiad with: `helmfile -e eqiad --state-values-set roll_restart=1 sync` [analytics]
10:31 <btullis> beginning hdfs datanode rolling restart with `sudo cumin -b 2 -p 80 -s 120 A:hadoop-worker 'systemctl restart hadoop-hdfs-datanode'` [analytics]
10:10 <btullis> btullis@an-master1001:~$ sudo systemctl start hadoop-hdfs-namenode [analytics]
10:00 <btullis> roll-restarting journal nodes with 30 seconds between each one: `sudo cumin -b 1 -p 100 -s 30 A:hadoop-hdfs-journal 'systemctl restart hadoop-hdfs-journalnode'` [analytics]
09:29 <btullis> restarting the yarn restart with `sudo cumin -b 5 -p 80 -s 30 A:hadoop-worker 'systemctl restart hadoop-yarn-nodemanager'` [analytics]
08:57 <btullis> executing `cookbook sre.hadoop.roll-restart-workers analytics` [analytics]
2023-07-03 §
12:52 <btullis> restarting the aqs service to pick up mediawiki history snapshot for June [analytics]
2023-06-29 §
13:44 <btullis> upgrading airflow on an-launcher1002 to version 2.6.1 [analytics]
2023-06-28 §
13:25 <btullis> upgrading an-test-worker1003 to bullseye, after upgrading firmware [analytics]
13:08 <btullis> upgrading idrac firmware of an-test-worker1003 via the cookbook for T329363 [analytics]
2023-06-27 §
14:53 <mforns> deployed airflow analytics to unbreak DataHub's Druid ingestion [analytics]
13:32 <joal> Rerun druid_load_pageviews_hourly_aggregated_daily after deploy [analytics]
13:32 <joal> druid_load_pageviews_hourly_aggregated_dailyRerun [analytics]
13:25 <joal> Deploy Airflow [analytics]
11:10 <joal> Deploy refinery onto HDFS [analytics]
11:01 <stevemunene> upgrading an-test-worker1003 to bullseye, keeping `/srv/hadoop` intact [analytics]
10:55 <joal> Deploy refinery using scap [analytics]
09:42 <stevemunene> !log run puppet on hadoop-masters this does a refresh of the hdfs nodes [analytics]
09:38 <stevemunene> Exclude analytics1061_1069 from HDFS and YARN [analytics]
09:21 <btullis> upgrading an-test-worker1002 to bullseye, keeping `/srv/hadoop` intact [analytics]
08:38 <elukey> revoked puppet cert for 'varnishkafka' and cleaned up its cergen's files in puppet private [analytics]
07:14 <elukey> `sudo kill `pgrep -u paramd`` on stat1005 to unblock puppet [analytics]
2023-06-26 §
23:22 <btullis> shutting down an-worker1092 in preparation for RAID controller battery replacement [analytics]
14:06 <elukey> move varnishkafka instances in esams to pki - T337825 [analytics]
11:39 <stevemunene> running hdfs dfsadmin -refreshNodes to pick up analytics106[1-3] from hosts.exclude [analytics]
11:35 <stevemunene> disable puppet on an-master1001.eqiad.wmnet [analytics]
09:40 <joal> Rerun failed druid-loading airflow jobs [analytics]
09:38 <btullis> deploying presto version 0.281 to production [analytics]
06:28 <stevemunene> run puppet on hadoop-masters [analytics]
06:27 <stevemunene> Excluding analytics106[4-6] from HDFS and YARN as we Decommission them [analytics]
2023-06-23 §
12:40 <elukey> move varnishkafka drmrs instances to pki - T337825 [analytics]
10:20 <btullis> reboot an-worker1110 after initializing a second replacement drive for T336929 [analytics]
10:16 <elukey> restart turnilo to pick up config changes - T340097 [analytics]
2023-06-22 §
15:57 <btullis> adding new bigtop-1.5 packages to apt.wikimedia.org for bullseye [analytics]
15:50 <elukey> update the webrequest_sampled_live druid kafka supervisor to add the https field - T340097 [analytics]
15:18 <btullis> cleared status for aqs_hourly.wait_for_webrequest run 13:00 and the downstream task on an-test-client1001. [analytics]
15:07 <btullis> clearing task for refine_webrequest_hourly_test_text hour 13:00 [analytics]
14:36 <btullis> restarted airflow-webserver and airflow-scheduler on an-test-client1001 with version 2.6.1. [analytics]
14:11 <btullis> redeploying datahub to staging to try to get upgrade to 0.10.0 working. [analytics]
14:02 <stevemunene> running sre.hadoop.roll-restart-masters restart the Namenodes to completely remove any reference of analytics106[1-3] T317861 [analytics]