|
2021-04-21
ยง
|
| 08:46 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores1005.eqiad.wmnet |
[production] |
| 08:46 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores1006.eqiad.wmnet |
[production] |
| 08:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 25%: Repool db1074', diff saved to https://phabricator.wikimedia.org/P15493 and previous config saved to /var/cache/conftool/dbconfig/20210421-084555-root.json |
[production] |
| 08:41 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores1006.eqiad.wmnet |
[production] |
| 08:41 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores1005.eqiad.wmnet |
[production] |
| 08:40 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores1004.eqiad.wmnet |
[production] |
| 08:40 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores1003.eqiad.wmnet |
[production] |
| 08:33 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores1004.eqiad.wmnet |
[production] |
| 08:33 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores1003.eqiad.wmnet |
[production] |
| 08:30 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores1002.eqiad.wmnet |
[production] |
| 08:22 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores1001.eqiad.wmnet |
[production] |
| 08:16 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores1002.eqiad.wmnet |
[production] |
| 08:16 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores1001.eqiad.wmnet |
[production] |
| 08:10 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2009.codfw.wmnet |
[production] |
| 08:05 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores2009.codfw.wmnet |
[production] |
| 08:05 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2008.codfw.wmnet |
[production] |
| 08:05 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2007.codfw.wmnet |
[production] |
| 07:59 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores2008.codfw.wmnet |
[production] |
| 07:58 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores2007.codfw.wmnet |
[production] |
| 07:58 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2006.codfw.wmnet |
[production] |
| 07:58 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2005.codfw.wmnet |
[production] |
| 07:53 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores2006.codfw.wmnet |
[production] |
| 07:52 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores2005.codfw.wmnet |
[production] |
| 07:52 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-coord1001.eqiad.wmnet with reason: REIMAGE |
[production] |
| 07:52 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2003.codfw.wmnet |
[production] |
| 07:52 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2004.codfw.wmnet |
[production] |
| 07:50 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-coord1001.eqiad.wmnet with reason: REIMAGE |
[production] |
| 07:46 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores2004.codfw.wmnet |
[production] |
| 07:46 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores2003.codfw.wmnet |
[production] |
| 07:44 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2002.codfw.wmnet |
[production] |
| 07:44 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2001.codfw.wmnet |
[production] |
| 07:39 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores2002.codfw.wmnet |
[production] |
| 07:38 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ores2001.codfw.wmnet |
[production] |
| 06:49 |
<akosiaris@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 06:49 |
<akosiaris@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 06:42 |
<elukey> |
upload hue_4.9.0-2+deb10u1 to buster-wikimedia |
[production] |
| 06:11 |
<marostegui> |
Stop MySQL on db1074 to clone db1156 (there will be lag in s2 in wiki replicas) T258361 |
[production] |
| 06:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1074 to clone db1156 T258361', diff saved to https://phabricator.wikimedia.org/P15491 and previous config saved to /var/cache/conftool/dbconfig/20210421-061019-marostegui.json |
[production] |
| 06:07 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2082.codfw.wmnet with reason: REIMAGE |
[production] |
| 06:05 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2077.codfw.wmnet with reason: REIMAGE |
[production] |
| 06:03 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2082.codfw.wmnet with reason: REIMAGE |
[production] |
| 06:03 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2077.codfw.wmnet with reason: REIMAGE |
[production] |
| 05:42 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1086.eqiad.wmnet |
[production] |
| 05:33 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts db1086.eqiad.wmnet |
[production] |
| 00:38 |
<pt1979@cumin2001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudnet2004-dev.codfw.wmnet with reason: REIMAGE |
[production] |
| 00:36 |
<pt1979@cumin2001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet2004-dev.codfw.wmnet with reason: REIMAGE |
[production] |
| 00:15 |
<ryankemper> |
[WDQS] Pooled `wdqs1003` |
[production] |
| 00:14 |
<ryankemper> |
[WDQS] Pooled `wdqs2008` |
[production] |
| 00:06 |
<ryankemper> |
`sudo -i wmf-auto-reimage-host -p T280382 wdqs1006.eqiad.wmnet` |
[production] |
| 00:04 |
<ryankemper> |
[WDQS] pooled `wdqs1004` |
[production] |