51-100 of 4807 results (23ms)
2022-10-10 §
15:36 <mforns> reran geoeditors_public_monthly airflow DAG for Sept 2022, after fix [analytics]
15:34 <mforns> deployed airflow to fix geoeditors_public_monthly DAG [analytics]
15:31 <mforns> started unique devices daily back-filling in cassandra from 1st of July to end of Sept [analytics]
2022-10-08 §
11:48 <joal> rerun webrequest-load-wf-text-2022-10-7-20 [analytics]
2022-10-07 §
09:26 <elukey> delete calico pods in CrashLoop on dse (probably due to the incorrect docker settings) [analytics]
07:54 <elukey> re-initialize docker on dse-k8s-worker1004 - wrong storage type set (devicemapper instead of overlay2) [analytics]
07:49 <elukey> re-initialize docker on dse-k8s-worker100[5-8] - wrong storage type set (devicemapper instead of overlay2) [analytics]
2022-10-06 §
19:51 <SandraEbele> Started airflow projectview_hourly_dag [analytics]
19:51 <SandraEbele> Killed Oozie projectview-hourly job [analytics]
19:40 <SandraEbele> Deployed airflow to fix projectview_hourly_dag [analytics]
13:48 <btullis> decommission aqs1007 (also forgot to log aqs1006) [analytics]
12:15 <btullis> decommissioning aqs1005 [analytics]
11:23 <btullis> decommissioning aqs1004 [analytics]
2022-10-05 §
16:48 <btullis> forcibly and lazily unmounted legacy labstore hosts from an-launcher1002 and removed their /etc/fstab entries [analytics]
15:27 <SandraEbele> deployed refinery source [analytics]
14:33 <mforns> finished refinery deploy - regular weekly train [analytics]
14:05 <mforns> starting refinery deploy - regular weekly train [analytics]
13:49 <SandraEbele> Started Airflow projectview_geo job [analytics]
13:48 <SandraEbele> killed Oozie projectview-geo-coord job [analytics]
13:21 <SandraEbele> deploying fix for projective tags on airflow. [analytics]
2022-10-04 §
09:53 <btullis> deployed eventgate-logging-external to eqiad (a few minutes ago) [analytics]
09:45 <btullis> deploying new eventgate-logging-external service to codfw [analytics]
09:44 <btullis> deploying new eventgate-logging-external service to staging [analytics]
2022-10-02 §
08:13 <elukey> apt-get clean on an-airflow1001 to free some space on the root partition [analytics]
2022-09-30 §
08:41 <btullis> restarted hive-server2 and hive-metastore services on an-coord1002 (standby) server [analytics]
2022-09-29 §
12:34 <joal> Rerun failed oozie webrequest-load-wf-text-2022-9-29-9 [analytics]
06:38 <joal> Try to rerun airflow unique_devices_daily.compute_per_project_family_metrics.2022-09-15 [analytics]
06:37 <joal> Rerun airflow unique_devices_dailyschedule: @daily [analytics]
2022-09-28 §
19:50 <mforns> killed oozie's unique_devices-per_domain-daily-coord because we migrated it to airflow [analytics]
19:49 <mforns> killed oozie's unique_devices-per_project_family-daily-coord because we migrated it to airflow [analytics]
19:48 <mforns> killed oozie's unique_devices-per_project_family-monthly-coord because we migrated it to airflow [analytics]
19:48 <mforns> killed oozie's unique_devices-per_domain-monthly-coord because we migrated it to airflow [analytics]
18:22 <mforns> deployed airflow to fix unique_devices jobs [analytics]
15:29 <SandraEbele> started airflow projectview_geo job [analytics]
15:01 <btullis> roll-restarting druid-analytics [analytics]
15:00 <SandraEbele> deploying Airflow for hdfsarchiver operator fix [analytics]
14:02 <btullis> roll-restarting druid-public [analytics]
09:22 <btullis> started cookbook sre.kafka.roll-restart-brokers jumbo-eqiad [analytics]
2022-09-27 §
15:05 <mforns> re-ran wikidata_metrics_to_graphite_daily failed airflow tasks [analytics]
15:03 <mforns> re-ran cassandra_daily_load failed airflow tasks [analytics]
14:59 <mforns> re-ran apis_metrics_to_graphite_hourly [analytics]
14:56 <mforns> deployed Airflow (fixed) [analytics]
14:23 <mforns> rolled back Airflow [analytics]
14:23 <mforns> deployed Airflow for 3 fixes [analytics]
2022-09-26 §
20:07 <xcollazo> Kill oozie geoeditors jobs for load, public monthly, and yearly after Airflow migration. [analytics]
16:13 <joal> rerunning failed webrequest-text-2022-09-26-15 [analytics]
13:48 <aqu> Deploying airflow-dags on analytics & analytics_test [analytics]
11:03 <btullis> failing back hive to an-coord1001 using DNS https://gerrit.wikimedia.org/r/c/operations/dns/+/832294 [analytics]
09:41 <btullis> rebooted matomo1002 at the VM level to pick up new disk [analytics]
09:40 <btullis> merged the spark3 patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/834500 [analytics]