1901-1950 of 10000 results (128ms)
2024-10-29 ยง
14:52 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
14:47 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host ganeti2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
14:44 <elukey@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
14:40 <reedy@deploy2002> Finished scap sync-world: 1.44.0-wmf.1 backports to fix deprecated logspam T375660 T377521 (duration: 07m 21s) [production]
14:39 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2038.codfw.wmnet [production]
14:39 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P70616 and previous config saved to /var/cache/conftool/dbconfig/20241029-143912-ladsgroup.json [production]
14:39 <herron> centrallog1002:~# systemctl restart rsyslogd [production]
14:38 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host dse-k8s-worker1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
14:35 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2038.codfw.wmnet [production]
14:35 <elukey@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
14:34 <elukey@cumin1002> START - Cookbook sre.hosts.reimage for host aux-k8s-worker1003.eqiad.wmnet with OS bookworm [production]
14:32 <reedy@deploy2002> Started scap sync-world: 1.44.0-wmf.1 backports to fix deprecated logspam T375660 T377521 [production]
14:29 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host ganeti2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
14:29 <elukey@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
14:25 <MichaelG_WMF> T372337 clearing dangling database-records for link suggestions by running `mwscript extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=eswiki --db-table --force` [production]
14:24 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host ganeti2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
14:24 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2116 (T376905)', diff saved to https://phabricator.wikimedia.org/P70615 and previous config saved to /var/cache/conftool/dbconfig/20241029-142405-ladsgroup.json [production]
14:20 <elukey@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-lab1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
14:19 <elukey> restart rsyslog on centrallog1002 - connection errors, failing prometheus probes [production]
14:18 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2037.codfw.wmnet [production]
14:18 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2037.codfw.wmnet [production]
14:17 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host ml-lab1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
14:16 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2116 (T376905)', diff saved to https://phabricator.wikimedia.org/P70614 and previous config saved to /var/cache/conftool/dbconfig/20241029-141532-ladsgroup.json [production]
14:16 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance [production]
14:15 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance [production]
14:14 <elukey@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
14:09 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host ganeti2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
14:07 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-lab1001.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
14:06 <kostajh> UTC afternoon deploys done [production]
14:05 <kharlan@deploy2002> Finished scap sync-world: Backport for [[gerrit:1084112|AuthManagerStatsdHandler: Add label for wiki (T375505)]], [[gerrit:1084111|AuthManagerStatsdHandler: Add label for wiki (T375505)]] (duration: 07m 53s) [production]
14:01 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host ml-lab1001.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
14:00 <kharlan@deploy2002> kharlan: Continuing with sync [production]
13:59 <kharlan@deploy2002> kharlan: Backport for [[gerrit:1084112|AuthManagerStatsdHandler: Add label for wiki (T375505)]], [[gerrit:1084111|AuthManagerStatsdHandler: Add label for wiki (T375505)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:57 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2036.codfw.wmnet [production]
13:57 <kharlan@deploy2002> Started scap sync-world: Backport for [[gerrit:1084112|AuthManagerStatsdHandler: Add label for wiki (T375505)]], [[gerrit:1084111|AuthManagerStatsdHandler: Add label for wiki (T375505)]] [production]
13:56 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2036.codfw.wmnet [production]
13:48 <jforrester@deploy2002> Finished scap sync-world: Backport for [[gerrit:1084110|fix ibawiki's tagline svg path]] (duration: 07m 41s) [production]
13:47 <ayounsi@cumin1002> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 16347 [production]
13:46 <ayounsi@cumin1002> START - Cookbook sre.network.peering with action 'configure' for AS: 16347 [production]
13:45 <ayounsi@cumin1002> END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 16347 [production]
13:45 <ayounsi@cumin1002> START - Cookbook sre.network.peering with action 'email' for AS: 16347 [production]
13:43 <jforrester@deploy2002> jforrester, hamishz: Continuing with sync [production]
13:42 <jforrester@deploy2002> jforrester, hamishz: Backport for [[gerrit:1084110|fix ibawiki's tagline svg path]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:42 <moritzm> installing ghoscript security updates [production]
13:40 <jforrester@deploy2002> Started scap sync-world: Backport for [[gerrit:1084110|fix ibawiki's tagline svg path]] [production]
13:38 <jforrester@deploy2002> Finished scap sync-world: Backport for [[gerrit:1082878|Allow admins on testwiki to grant and remove upwizcampeditors (T378067)]], [[gerrit:1082444|nlwiki, commonswiki, wikidata: lift IP cap for edit-a-thon (T377930)]] (duration: 08m 03s) [production]
13:34 <jforrester@deploy2002> dreamrimmer, superzerocool, jforrester: Continuing with sync [production]
13:33 <jforrester@deploy2002> dreamrimmer, superzerocool, jforrester: Backport for [[gerrit:1082878|Allow admins on testwiki to grant and remove upwizcampeditors (T378067)]], [[gerrit:1082444|nlwiki, commonswiki, wikidata: lift IP cap for edit-a-thon (T377930)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:31 <jforrester@deploy2002> Started scap sync-world: Backport for [[gerrit:1082878|Allow admins on testwiki to grant and remove upwizcampeditors (T378067)]], [[gerrit:1082444|nlwiki, commonswiki, wikidata: lift IP cap for edit-a-thon (T377930)]] [production]
13:31 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2211 (re)pooling @ 100%: post clone repool', diff saved to https://phabricator.wikimedia.org/P70612 and previous config saved to /var/cache/conftool/dbconfig/20241029-132956-arnaudb.json [production]