5901-5950 of 10000 results (99ms)
2024-06-05 ยง
13:21 <fabfur> enable magru DC after applying IPIP encapsulation patches (T366466) [production]
13:20 <mvernon@cumin2002> START - Cookbook sre.hosts.reboot-single for host ms-be2071.codfw.wmnet [production]
13:19 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2070.codfw.wmnet [production]
13:17 <dreamyjazz@deploy1002> dreamyjazz: Continuing with sync [production]
13:17 <dreamyjazz@deploy1002> dreamyjazz: Backport for [[gerrit:1038839|Follow-up: Don't run interact with block buttons if they don't exist (T329493)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:16 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2179 (T364299)', diff saved to https://phabricator.wikimedia.org/P64104 and previous config saved to /var/cache/conftool/dbconfig/20240605-131647-marostegui.json [production]
13:16 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2179.codfw.wmnet with reason: Maintenance [production]
13:16 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db2179.codfw.wmnet with reason: Maintenance [production]
13:16 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2172 (T364299)', diff saved to https://phabricator.wikimedia.org/P64103 and previous config saved to /var/cache/conftool/dbconfig/20240605-131623-marostegui.json [production]
13:14 <dreamyjazz@deploy1002> Started scap: Backport for [[gerrit:1038839|Follow-up: Don't run interact with block buttons if they don't exist (T329493)]] [production]
13:13 <mvernon@cumin2002> START - Cookbook sre.hosts.reboot-single for host ms-be2070.codfw.wmnet [production]
13:13 <mvernon@cumin1002> START - Cookbook sre.hosts.reboot-single for host ms-be1071.eqiad.wmnet [production]
13:13 <dreamyjazz@deploy1002> Finished scap: Backport for [[gerrit:1013386|[CheckUser] Stop writing old for event table migration on testwiki (T360686)]] (duration: 19m 13s) [production]
13:10 <elukey@cumin1002> END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker [production]
13:06 <fabfur> restarting pybal on lvs7001/lvs7003 to appy IPIP conf (T366466) [production]
13:04 <dreamyjazz@deploy1002> dreamyjazz: Continuing with sync [production]
13:03 <mvernon@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1070.eqiad.wmnet [production]
13:02 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2069.codfw.wmnet [production]
13:02 <dreamyjazz@deploy1002> dreamyjazz: Backport for [[gerrit:1013386|[CheckUser] Stop writing old for event table migration on testwiki (T360686)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
13:01 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P64102 and previous config saved to /var/cache/conftool/dbconfig/20240605-130115-marostegui.json [production]
12:56 <elukey@cumin1002> START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker [production]
12:55 <mvernon@cumin1002> START - Cookbook sre.hosts.reboot-single for host ms-be1070.eqiad.wmnet [production]
12:55 <mvernon@cumin2002> START - Cookbook sre.hosts.reboot-single for host ms-be2069.codfw.wmnet [production]
12:53 <dreamyjazz@deploy1002> Started scap: Backport for [[gerrit:1013386|[CheckUser] Stop writing old for event table migration on testwiki (T360686)]] [production]
12:53 <mvernon@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1069.eqiad.wmnet [production]
12:52 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host ping2004.codfw.wmnet [production]
12:52 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ping2004.codfw.wmnet with OS bookworm [production]
12:51 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2068.codfw.wmnet [production]
12:49 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1246.eqiad.wmnet with reason: maintenance [production]
12:49 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on db1246.eqiad.wmnet with reason: maintenance [production]
12:49 <arnaudb@cumin1002> dbctl commit (dc=all): 'depool db1246 T363119', diff saved to https://phabricator.wikimedia.org/P64101 and previous config saved to /var/cache/conftool/dbconfig/20240605-124918-arnaudb.json [production]
12:46 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P64100 and previous config saved to /var/cache/conftool/dbconfig/20240605-124607-marostegui.json [production]
12:46 <mvernon@cumin1002> START - Cookbook sre.hosts.reboot-single for host ms-be1069.eqiad.wmnet [production]
12:45 <mvernon@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1068.eqiad.wmnet [production]
12:45 <mvernon@cumin2002> START - Cookbook sre.hosts.reboot-single for host ms-be2068.codfw.wmnet [production]
12:45 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2067.codfw.wmnet [production]
12:43 <moritzm> failover ganeti masters in drmrs [production]
12:40 <cgoubert@cumin1002> END (ERROR) - Cookbook sre.k8s.reboot-nodes (exit_code=97) rolling reboot on A:wikikube-worker-codfw [production]
12:39 <mvernon@cumin1002> START - Cookbook sre.hosts.reboot-single for host ms-be1068.eqiad.wmnet [production]
12:39 <mvernon@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1067.eqiad.wmnet [production]
12:38 <mvernon@cumin2002> START - Cookbook sre.hosts.reboot-single for host ms-be2067.codfw.wmnet [production]
12:38 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2066.codfw.wmnet [production]
12:36 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ping2004.codfw.wmnet with reason: host reimage [production]
12:35 <fabfur> disabling puppet on A:cp-text to test IPIP encapsulation on magru (T366466) [production]
12:33 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet [production]
12:33 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet [production]
12:32 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ping2004.codfw.wmnet with reason: host reimage [production]
12:32 <mvernon@cumin2002> START - Cookbook sre.hosts.reboot-single for host ms-be2066.codfw.wmnet [production]
12:31 <mvernon@cumin1002> START - Cookbook sre.hosts.reboot-single for host ms-be1067.eqiad.wmnet [production]
12:31 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2172 (T364299)', diff saved to https://phabricator.wikimedia.org/P64099 and previous config saved to /var/cache/conftool/dbconfig/20240605-123059-marostegui.json [production]