5001-5050 of 10000 results (66ms)
2022-06-14 ยง
14:15 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2011.codfw.wmnet with OS buster [production]
14:14 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2008.codfw.wmnet with OS buster [production]
14:13 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4002.ulsfo.wmnet [production]
14:12 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2007.codfw.wmnet with OS buster [production]
14:10 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2006.codfw.wmnet with OS buster [production]
14:09 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti4002.ulsfo.wmnet [production]
14:06 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid2002.codfw.wmnet [production]
14:04 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host failoid2002.codfw.wmnet [production]
13:54 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2005.codfw.wmnet with OS buster [production]
13:27 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
13:27 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
13:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1157 (T310011)', diff saved to https://phabricator.wikimedia.org/P29741 and previous config saved to /var/cache/conftool/dbconfig/20220614-132654-marostegui.json [production]
13:13 <urbanecm> UTC afternoon B&C window done [production]
13:12 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 1692de09bf04c724cf416679405d4b6485550d40: Disable DiscussionTools visualenhancements feature in production (duration: 03m 25s) [production]
13:11 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P29740 and previous config saved to /var/cache/conftool/dbconfig/20220614-131149-marostegui.json [production]
13:11 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
13:10 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
13:09 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
13:09 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2011.codfw.wmnet with reason: host reimage [production]
13:09 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
13:08 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 7f2dc7296f0c25d00e45651c50c3e45733cc63b3: Make new topic tool available as opt-out almost everywhere (phrase 4; T310392) (duration: 03m 45s) [production]
13:06 <mvernon@cumin2002> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on aqs2012.codfw.wmnet with reason: host reimage [production]
13:06 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2010.codfw.wmnet with reason: host reimage [production]
13:04 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2012.codfw.wmnet with reason: host reimage [production]
13:04 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2009.codfw.wmnet with reason: host reimage [production]
13:02 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2011.codfw.wmnet with reason: host reimage [production]
13:02 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2008.codfw.wmnet with reason: host reimage [production]
13:01 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2010.codfw.wmnet with reason: host reimage [production]
13:01 <mvernon@cumin2002> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on aqs2007.codfw.wmnet with reason: host reimage [production]
12:59 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2006.codfw.wmnet with reason: host reimage [production]
12:59 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2009.codfw.wmnet with reason: host reimage [production]
12:57 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2008.codfw.wmnet with reason: host reimage [production]
12:57 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2007.codfw.wmnet with reason: host reimage [production]
12:57 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs2005.codfw.wmnet with reason: host reimage [production]
12:56 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P29739 and previous config saved to /var/cache/conftool/dbconfig/20220614-125644-marostegui.json [production]
12:56 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2006.codfw.wmnet with reason: host reimage [production]
12:53 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2005.codfw.wmnet with reason: host reimage [production]
12:47 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host aqs2012.codfw.wmnet with OS buster [production]
12:46 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host aqs2011.codfw.wmnet with OS buster [production]
12:45 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host aqs2010.codfw.wmnet with OS buster [production]
12:42 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host aqs2009.codfw.wmnet with OS buster [production]
12:41 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host aqs2008.codfw.wmnet with OS buster [production]
12:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1157 (T310011)', diff saved to https://phabricator.wikimedia.org/P29738 and previous config saved to /var/cache/conftool/dbconfig/20220614-124139-marostegui.json [production]
12:40 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host aqs2007.codfw.wmnet with OS buster [production]
12:40 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aqs2004.codfw.wmnet with OS buster [production]
12:39 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host aqs2006.codfw.wmnet with OS buster [production]
12:38 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host aqs2005.codfw.wmnet with OS buster [production]
12:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1157 (T310011)', diff saved to https://phabricator.wikimedia.org/P29737 and previous config saved to /var/cache/conftool/dbconfig/20220614-120323-marostegui.json [production]
12:03 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]
12:03 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1157.eqiad.wmnet with reason: Maintenance [production]