2651-2700 of 10000 results (86ms)
2023-09-19 ยง
14:20 <oblivian@deploy1002> Started scap: (no justification provided) [production]
14:20 <kamila@deploy1002> Unlocked for deployment [ALL REPOSITORIES]: Datacenter Switchover: Services & Traffic - T346330 (duration: 19m 27s) [production]
14:01 <kamila@cumin1001> START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter Switchover: Services - T346330 [production]
14:00 <kamila@deploy1002> Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover: Services & Traffic - T346330 [production]
13:58 <TheresNoTime> `[samtar@mwmaint1002 ~]$ mwscript namespaceDupes.php --wiki shwiki --fix` T346588 [production]
13:57 <samtar@deploy1002> Finished scap: Backport for [[gerrit:958421|Add namespace aliases to shwiki (T346588)]] (duration: 51m 50s) [production]
13:53 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:53 <stevemunene@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-test-client1001.eqiad.wmnet [production]
13:53 <stevemunene@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:52 <elukey> clean old puppet certs kafka_logging-eqiad_broker [production]
13:52 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
13:52 <jhancock@cumin2002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
13:52 <jhancock@cumin2002> END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Updating DNS record of kuberbetes2026 - jhancock@cumin2002" [production]
13:51 <stevemunene@cumin1001> START - Cookbook sre.dns.netbox [production]
13:50 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Updating DNS record of kuberbetes2026 - jhancock@cumin2002" [production]
13:47 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
13:47 <jebe@deploy1002> Finished deploy [airflow-dags/analytics@6b9855a]: (no justification provided) (duration: 00m 43s) [production]
13:46 <stevemunene@cumin1001> START - Cookbook sre.hosts.decommission for hosts an-test-client1001.eqiad.wmnet [production]
13:46 <jebe@deploy1002> Started deploy [airflow-dags/analytics@6b9855a]: (no justification provided) [production]
13:34 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flerovium.eqiad.wmnet [production]
13:33 <samtar@deploy1002> samtar and aleksandar: Continuing with sync [production]
13:28 <samtar@deploy1002> samtar and aleksandar: Backport for [[gerrit:958421|Add namespace aliases to shwiki (T346588)]] synced to the testservers mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) [production]
13:27 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host flerovium.eqiad.wmnet [production]
13:17 <jebe@deploy1002> Finished deploy [analytics/refinery@2d9d6d0] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2d9d6d0] (duration: 02m 06s) [production]
13:15 <Emperor> ms-be10[44-60] swift package updates T346730 [production]
13:15 <jebe@deploy1002> Started deploy [analytics/refinery@2d9d6d0] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2d9d6d0] [production]
13:14 <jebe@deploy1002> Finished deploy [analytics/refinery@2d9d6d0] (thin): Regular analytics weekly train THIN [analytics/refinery@2d9d6d0] (duration: 00m 04s) [production]
13:14 <jebe@deploy1002> Started deploy [analytics/refinery@2d9d6d0] (thin): Regular analytics weekly train THIN [analytics/refinery@2d9d6d0] [production]
13:14 <jebe@deploy1002> Finished deploy [analytics/refinery@2d9d6d0]: Regular analytics weekly train [analytics/refinery@2d9d6d0] (duration: 05m 52s) [production]
13:08 <jebe@deploy1002> Started deploy [analytics/refinery@2d9d6d0]: Regular analytics weekly train [analytics/refinery@2d9d6d0] [production]
13:05 <samtar@deploy1002> Started scap: Backport for [[gerrit:958421|Add namespace aliases to shwiki (T346588)]] [production]
13:05 <btullis@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-jumbo-eqiad cluster: Roll restart of jvm daemons. [production]
12:44 <Emperor> ms-be20[60-73] swift package updates T346730 [production]
12:22 <Emperor> ms-be20[49-59] swift package updates T346730 [production]
12:19 <jebe@deploy1002> Finished deploy [analytics/refinery@91bb4a0] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@91bb4a0] (duration: 02m 03s) [production]
12:18 <Emperor> ms-be2048 swift package updates T346730 [production]
12:17 <jebe@deploy1002> Started deploy [analytics/refinery@91bb4a0] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@91bb4a0] [production]
12:17 <jebe@deploy1002> Finished deploy [analytics/refinery@91bb4a0] (thin): Regular analytics weekly train THIN [analytics/refinery@91bb4a0] (duration: 00m 05s) [production]
12:17 <jebe@deploy1002> Started deploy [analytics/refinery@91bb4a0] (thin): Regular analytics weekly train THIN [analytics/refinery@91bb4a0] [production]
12:14 <Emperor> ms-be2047 swift package updates T346730 [production]
12:12 <Emperor> ms-be204{5,6} swift package updates T346730 [production]
12:10 <jebe@deploy1002> Finished deploy [analytics/refinery@91bb4a0]: Regular analytics weekly train [analytics/refinery@91bb4a0] (duration: 06m 53s) [production]
12:08 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
12:03 <jebe@deploy1002> Started deploy [analytics/refinery@91bb4a0]: Regular analytics weekly train [analytics/refinery@91bb4a0] [production]
11:51 <ayounsi@cumin1001> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox [production]
11:51 <ayounsi@cumin1001> START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox [production]
11:50 <ayounsi@cumin1001> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary [production]
11:48 <ayounsi@cumin1001> START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary [production]
11:21 <marostegui@cumin1001> dbctl commit (dc=all): 'db1134 (re)pooling @ 100%: Repooling after recloning db1128', diff saved to https://phabricator.wikimedia.org/P52530 and previous config saved to /var/cache/conftool/dbconfig/20230919-112156-root.json [production]
11:09 <Emperor> eqiad swift front-end swift package updates T346730 [production]