1401-1450 of 6005 results (29ms)
2022-07-07 §
09:02 <btullis> restarting dbstore1003 as per announced maintenance window [analytics]
2022-07-06 §
18:09 <ottomata> enabling iceberg hive catalog connector on analytics_cluster presto [analytics]
17:57 <ottomata> upgrading presto to 0.273.3 in analytics cluster - T311525 [analytics]
09:50 <btullis> roll-restarting hadoop workers on the test cluster. [analytics]
09:46 <btullis> restarting refinery-drop-webrequest-raw-partitions.service on an-test-coord1001 [analytics]
09:44 <btullis> restarting refinery-drop-webrequest-refined-partitions.service on an-test-coord1001 [analytics]
09:42 <btullis> restarted drop_event.service on an-test-coord1001 [analytics]
09:35 <btullis> restarting hive-server2 and hive-metastore on an-test-coord1001 [analytics]
2022-07-05 §
11:01 <btullis> sudo cookbook sre.hadoop.roll-restart-masters test [analytics]
2022-07-04 §
16:14 <btullis> systemctl restart airflow-scheduler@research.service (on an-airflow1002) [analytics]
08:04 <elukey> kill leftover processes of user `mewoph` on stat100x to allow puppet runs [analytics]
2022-06-29 §
17:27 <mforns> killed mediawiki-history-load bundle in Hue, and started corresponding mediawiki_history_load DAG in Airflow [analytics]
13:12 <mforns> re-deployed refinery with scap and refinery-deploy-to-hdfs [analytics]
11:51 <btullis> btullis@an-master1001:~$ sudo -u hdfs /usr/bin/hdfs haadmin -failover an-master1002-eqiad-wmnet an-master1001-eqiad-wmnet [analytics]
2022-06-28 §
20:57 <mforns> refinery deploy failed and I rolled back successfully, will try and repeat tomorrow when other people are present :] [analytics]
20:19 <mforns> starting refinery deployment for refinery-source v0.2.2 [analytics]
20:19 <mforns> starting refinery deploymenty [analytics]
17:25 <ottomata> installing presto 0.273.3 on an-test-coord1001 and an-test-presto1001 [analytics]
12:48 <milimetric> deploying airflow-dags/analytics to work on the metadata ingestion jobs [analytics]
2022-06-27 §
20:33 <btullis> systemctl reset-failed jupyter-aarora-singleuser and jupyter-seddon-singleuser on stat1005 [analytics]
20:16 <btullis> checking and restarting prometheus-mysqld-exporter on an-coord1001 [analytics]
15:25 <btullis> upgraded conda-base-env on an-test-client1001 from 0.0.1 to 0.0.4 [analytics]
2022-06-24 §
15:14 <ottomata> backfilled eventlogging data lost during failed gobblin job - T311263 [analytics]
2022-06-23 §
13:48 <btullis> started the namenode service on an-master1001 after failback failure [analytics]
13:41 <btullis> The failback didn't work again. [analytics]
13:39 <btullis> attempting failback of namenode service from an-master1002 to an-master1001 [analytics]
13:07 <btullis> restarted hadoop-hdfs-namenode service on an-master1001 [analytics]
11:25 <joal> kill oozie mediawiki-geoeditors-monthly-coord in favor of airflow job [analytics]
08:52 <joal> Deploy airflow [analytics]
2022-06-22 §
20:55 <aqu> `scap deploy -f analytics/refinery` because of a crash during `git-fat pull` [analytics]
19:30 <aqu> Deploying analytics/refinery [analytics]
2022-06-21 §
14:56 <aqu> RefineSanitize from an-launcher1002: sudo -u analytics kerberos-run-command analytics spark2-submit --class org.wikimedia.analytics.refinery.job.refine.RefineSanitize --master yarn --deploy-mode client /srv/deployment/analytics/refinery/artifacts/org/wikimedia/analytics/refinery/refinery-job-0.1.15.jar --config_file /home/aqu/refine.properties --since "2022-06-19T09:52:00+0000" --until [analytics]
13:33 <aqu> sudo systemctl start monitor_refine_event_sanitized_main_immediate.service on an-launcher1002 [analytics]
10:47 <btullis> proceeding with the hadoop.roll-restart-masters cookbook [analytics]
2022-06-20 §
07:14 <SandraEbele> Started Airflow 3 Wikidata metrics jobs (Articleplaceholder, Reliability and SpecialEntityData metrics). [analytics]
07:12 <SandraEbele> Started Airflow3 Wikidata metrics jobs (Articleplaceholder, Relia) [analytics]
07:11 <SandraEbele> killed Oozie wikidata-articleplaceholder_metrics-coord, wikidata-reliability_metrics-coord, and wikidata-specialentitydata_metrics-coord jobs. [analytics]
2022-06-17 §
12:35 <SandraEbele> deployed daily airflow dag for 3 Wikidata metrics. [analytics]
08:36 <btullis> power cycled an-worker1109 as it was stuck with CPU soft lockups [analytics]
2022-06-16 §
06:49 <joal> Rerun webrequest-load-wf-upload-2022-6-15-22 after weird oozie failure [analytics]
2022-06-15 §
14:48 <btullis> deploying datahub 0.8.38 [analytics]
2022-06-14 §
10:48 <joal> unpause renamed dags [analytics]
10:44 <joal> Deploy Airflow [analytics]
10:12 <btullis> manually failing back hdfs-namenode to an-master1001 after fixing typo [analytics]
09:36 <joal> deploy refinery onto HDFS [analytics]
08:48 <btullis> roll-restarting hadoop masters T310293 [analytics]
08:40 <joal> Deploying using scap again after failure cleanup on an-launcher1002 [analytics]
07:45 <joal> deploy refinery using scap [analytics]
2022-06-13 §
14:00 <btullis> restarting presto service on an-coord1001 [analytics]
13:20 <btullis> btullis@datahubsearch1001:~$ sudo systemctl reset-failed ifup@ens13.service T273026 [analytics]