3551-3600 of 10000 results (44ms)
2021-04-21 §
08:33 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ores1004.eqiad.wmnet [production]
08:33 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ores1003.eqiad.wmnet [production]
08:30 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores1002.eqiad.wmnet [production]
08:22 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores1001.eqiad.wmnet [production]
08:16 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ores1002.eqiad.wmnet [production]
08:16 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ores1001.eqiad.wmnet [production]
08:10 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2009.codfw.wmnet [production]
08:05 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ores2009.codfw.wmnet [production]
08:05 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2008.codfw.wmnet [production]
08:05 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2007.codfw.wmnet [production]
07:59 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ores2008.codfw.wmnet [production]
07:58 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ores2007.codfw.wmnet [production]
07:58 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2006.codfw.wmnet [production]
07:58 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2005.codfw.wmnet [production]
07:53 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ores2006.codfw.wmnet [production]
07:52 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ores2005.codfw.wmnet [production]
07:52 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-coord1001.eqiad.wmnet with reason: REIMAGE [production]
07:52 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2003.codfw.wmnet [production]
07:52 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2004.codfw.wmnet [production]
07:50 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-coord1001.eqiad.wmnet with reason: REIMAGE [production]
07:46 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ores2004.codfw.wmnet [production]
07:46 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ores2003.codfw.wmnet [production]
07:44 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2002.codfw.wmnet [production]
07:44 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2001.codfw.wmnet [production]
07:39 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ores2002.codfw.wmnet [production]
07:38 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host ores2001.codfw.wmnet [production]
06:49 <akosiaris@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
06:49 <akosiaris@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
06:42 <elukey> upload hue_4.9.0-2+deb10u1 to buster-wikimedia [production]
06:11 <marostegui> Stop MySQL on db1074 to clone db1156 (there will be lag in s2 in wiki replicas) T258361 [production]
06:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1074 to clone db1156 T258361', diff saved to https://phabricator.wikimedia.org/P15491 and previous config saved to /var/cache/conftool/dbconfig/20210421-061019-marostegui.json [production]
06:07 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2082.codfw.wmnet with reason: REIMAGE [production]
06:05 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2077.codfw.wmnet with reason: REIMAGE [production]
06:03 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db2082.codfw.wmnet with reason: REIMAGE [production]
06:03 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db2077.codfw.wmnet with reason: REIMAGE [production]
05:42 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1086.eqiad.wmnet [production]
05:33 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission for hosts db1086.eqiad.wmnet [production]
00:38 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudnet2004-dev.codfw.wmnet with reason: REIMAGE [production]
00:36 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet2004-dev.codfw.wmnet with reason: REIMAGE [production]
00:15 <ryankemper> [WDQS] Pooled `wdqs1003` [production]
00:14 <ryankemper> [WDQS] Pooled `wdqs2008` [production]
00:06 <ryankemper> `sudo -i wmf-auto-reimage-host -p T280382 wdqs1006.eqiad.wmnet` [production]
00:04 <ryankemper> [WDQS] pooled `wdqs1004` [production]
2021-04-20 §
23:46 <urbanecm@deploy1002> Synchronized dblists/growthexperiments.dblist: 73544ccb40d9687b54c039aceb05cd033901d86f: urwiki: Enable Growth team features in stealth mode (T280067) (duration: 00m 57s) [production]
23:44 <urbanecm@deploy1002> Synchronized wmf-config/config/urwiki.yaml: 73544ccb40d9687b54c039aceb05cd033901d86f: urwiki: Enable Growth team features in stealth mode (T280067) (duration: 00m 57s) [production]
23:42 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 73544ccb40d9687b54c039aceb05cd033901d86f: urwiki: Enable Growth team features in stealth mode (T280067) (duration: 00m 58s) [production]
23:38 <Urbanecm> [urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=urwiki GrowthExperiments # T280067 [production]
23:38 <urbanecm@deploy1002> Synchronized dblists/growthexperiments.dblist: 314367bca6e924136704911b55fd3e2c929fa704: elwiki: Enable Growth team features in stealth mode (T280172; 3/3) (duration: 00m 56s) [production]
23:36 <urbanecm@deploy1002> Synchronized wmf-config/config/elwiki.yaml: 314367bca6e924136704911b55fd3e2c929fa704: elwiki: Enable Growth team features in stealth mode (T280172; 2/3) (duration: 00m 57s) [production]
23:35 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 314367bca6e924136704911b55fd3e2c929fa704: elwiki: Enable Growth team features in stealth mode (T280172; 1/3) (duration: 00m 57s) [production]