4801-4850 of 10000 results (114ms)
2024-06-10 ยง
07:32 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2012.codfw.wmnet [production]
07:31 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2011.codfw.wmnet [production]
07:31 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2011.codfw.wmnet [production]
07:30 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti1009.eqiad.wmnet [production]
07:26 <jayme@deploy1002> helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply [production]
07:25 <jayme@deploy1002> helmfile [eqiad] START helmfile.d/services/push-notifications: apply [production]
07:24 <jayme@deploy1002> helmfile [codfw] DONE helmfile.d/services/push-notifications: apply [production]
07:23 <jayme@deploy1002> helmfile [codfw] START helmfile.d/services/push-notifications: apply [production]
07:23 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2011.codfw.wmnet [production]
07:22 <jayme@deploy1002> helmfile [staging] DONE helmfile.d/services/push-notifications: apply [production]
07:22 <jayme@deploy1002> helmfile [staging] START helmfile.d/services/push-notifications: apply [production]
07:20 <marostegui@cumin1002> dbctl commit (dc=all): 'db2218 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P64501 and previous config saved to /var/cache/conftool/dbconfig/20240610-072043-root.json [production]
07:17 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1009.eqiad.wmnet [production]
07:15 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2011.codfw.wmnet [production]
07:14 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2010.codfw.wmnet [production]
07:13 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2010.codfw.wmnet [production]
07:07 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2010.codfw.wmnet [production]
07:05 <marostegui@cumin1002> dbctl commit (dc=all): 'db2218 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P64500 and previous config saved to /var/cache/conftool/dbconfig/20240610-070537-root.json [production]
07:03 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2010.codfw.wmnet [production]
07:02 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1168 (T364069)', diff saved to https://phabricator.wikimedia.org/P64499 and previous config saved to /var/cache/conftool/dbconfig/20240610-070249-marostegui.json [production]
07:02 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1168.eqiad.wmnet with reason: Maintenance [production]
07:02 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1168.eqiad.wmnet with reason: Maintenance [production]
07:02 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1165 (T364069)', diff saved to https://phabricator.wikimedia.org/P64498 and previous config saved to /var/cache/conftool/dbconfig/20240610-070224-marostegui.json [production]
07:00 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2009.codfw.wmnet [production]
06:59 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2009.codfw.wmnet [production]
06:58 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance [production]
06:58 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance [production]
06:56 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2202.codfw.wmnet with reason: Maintenance [production]
06:56 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2202.codfw.wmnet with reason: Maintenance [production]
06:56 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2188 (T352010)', diff saved to https://phabricator.wikimedia.org/P64497 and previous config saved to /var/cache/conftool/dbconfig/20240610-065640-ladsgroup.json [production]
06:53 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2009.codfw.wmnet [production]
06:50 <marostegui@cumin1002> dbctl commit (dc=all): 'db2218 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P64496 and previous config saved to /var/cache/conftool/dbconfig/20240610-065031-root.json [production]
06:47 <marostegui> dbmaint codfw s4 deploy schema change on db2140 T364299 [production]
06:47 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P64495 and previous config saved to /var/cache/conftool/dbconfig/20240610-064716-marostegui.json [production]
06:46 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2140.codfw.wmnet with reason: Long schema change [production]
06:46 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2140.codfw.wmnet with reason: Long schema change [production]
06:45 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2009.codfw.wmnet [production]
06:41 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P64494 and previous config saved to /var/cache/conftool/dbconfig/20240610-064132-ladsgroup.json [production]
06:39 <arnaudb@cumin1002> dbctl commit (dc=all): 'Set db2207 with weight 0 T367019', diff saved to https://phabricator.wikimedia.org/P64493 and previous config saved to /var/cache/conftool/dbconfig/20240610-063912-arnaudb.json [production]
06:39 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2140 T367017', diff saved to https://phabricator.wikimedia.org/P64492 and previous config saved to /var/cache/conftool/dbconfig/20240610-063904-root.json [production]
06:38 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s2 T367019 [production]
06:38 <marostegui@cumin1002> dbctl commit (dc=all): 'Promote db2179 to s4 primary T367017', diff saved to https://phabricator.wikimedia.org/P64491 and previous config saved to /var/cache/conftool/dbconfig/20240610-063830-root.json [production]
06:38 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Primary switchover s2 T367019 [production]
06:38 <marostegui> Starting s4 codfw failover from db2140 to db2179 - T367017 [production]
06:35 <marostegui@cumin1002> dbctl commit (dc=all): 'db2218 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P64490 and previous config saved to /var/cache/conftool/dbconfig/20240610-063524-root.json [production]
06:32 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P64489 and previous config saved to /var/cache/conftool/dbconfig/20240610-063208-marostegui.json [production]
06:26 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P64488 and previous config saved to /var/cache/conftool/dbconfig/20240610-062624-ladsgroup.json [production]
06:20 <marostegui@cumin1002> dbctl commit (dc=all): 'db2218 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P64487 and previous config saved to /var/cache/conftool/dbconfig/20240610-062017-root.json [production]
06:19 <marostegui@cumin1002> dbctl commit (dc=all): 'Remove db2179 from API/vslow/dump T367017', diff saved to https://phabricator.wikimedia.org/P64486 and previous config saved to /var/cache/conftool/dbconfig/20240610-061939-root.json [production]
06:19 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s4 T367017 [production]