|
2022-11-22
ยง
|
| 09:36 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on graphite2004.codfw.wmnet with reason: setup |
[production] |
| 09:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1106 (T321130)', diff saved to https://phabricator.wikimedia.org/P40483 and previous config saved to /var/cache/conftool/dbconfig/20221122-093556-marostegui.json |
[production] |
| 09:35 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:35 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 10:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:35 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1106.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:35 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1106.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:31 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1050.eqiad.wmnet with reason: host reimage |
[production] |
| 09:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P40482 and previous config saved to /var/cache/conftool/dbconfig/20221122-093128-marostegui.json |
[production] |
| 09:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P40481 and previous config saved to /var/cache/conftool/dbconfig/20221122-093044-ladsgroup.json |
[production] |
| 09:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T321130)', diff saved to https://phabricator.wikimedia.org/P40480 and previous config saved to /var/cache/conftool/dbconfig/20221122-092845-marostegui.json |
[production] |
| 09:27 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1050.eqiad.wmnet with reason: host reimage |
[production] |
| 09:26 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P40479 and previous config saved to /var/cache/conftool/dbconfig/20221122-092618-ladsgroup.json |
[production] |
| 09:25 |
<moritzm> |
failover Ganeti master in eqiad to ganeti1028 T311687 |
[production] |
| 09:24 |
<oblivian@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/apertium: apply |
[production] |
| 09:23 |
<oblivian@deploy1002> |
helmfile [codfw] START helmfile.d/services/apertium: apply |
[production] |
| 09:22 |
<oblivian@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/apertium: apply |
[production] |
| 09:22 |
<oblivian@deploy1002> |
helmfile [codfw] START helmfile.d/services/apertium: apply |
[production] |
| 09:19 |
<oblivian@deploy1002> |
helmfile [staging] DONE helmfile.d/services/apertium: apply |
[production] |
| 09:18 |
<oblivian@deploy1002> |
helmfile [staging] START helmfile.d/services/apertium: apply |
[production] |
| 09:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1110', diff saved to https://phabricator.wikimedia.org/P40478 and previous config saved to /var/cache/conftool/dbconfig/20221122-091621-marostegui.json |
[production] |
| 09:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P40477 and previous config saved to /var/cache/conftool/dbconfig/20221122-091537-ladsgroup.json |
[production] |
| 09:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P40476 and previous config saved to /var/cache/conftool/dbconfig/20221122-091339-marostegui.json |
[production] |
| 09:13 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1050.eqiad.wmnet with OS bullseye |
[production] |
| 09:11 |
<jmm@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cumin2002.codfw.wmnet |
[production] |
| 09:11 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2120 (T322618)', diff saved to https://phabricator.wikimedia.org/P40475 and previous config saved to /var/cache/conftool/dbconfig/20221122-091112-ladsgroup.json |
[production] |
| 09:08 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2120 (T322618)', diff saved to https://phabricator.wikimedia.org/P40474 and previous config saved to /var/cache/conftool/dbconfig/20221122-090854-ladsgroup.json |
[production] |
| 09:08 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2120.codfw.wmnet with reason: Maintenance |
[production] |
| 09:08 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2120.codfw.wmnet with reason: Maintenance |
[production] |
| 09:08 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2108 (T322618)', diff saved to https://phabricator.wikimedia.org/P40473 and previous config saved to /var/cache/conftool/dbconfig/20221122-090833-ladsgroup.json |
[production] |
| 09:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1110 (T321126)', diff saved to https://phabricator.wikimedia.org/P40472 and previous config saved to /var/cache/conftool/dbconfig/20221122-090115-marostegui.json |
[production] |
| 09:00 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P40471 and previous config saved to /var/cache/conftool/dbconfig/20221122-090030-ladsgroup.json |
[production] |
| 09:00 |
<jmm@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host cumin2002.codfw.wmnet |
[production] |
| 08:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1110 (T321126)', diff saved to https://phabricator.wikimedia.org/P40470 and previous config saved to /var/cache/conftool/dbconfig/20221122-085843-marostegui.json |
[production] |
| 08:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P40469 and previous config saved to /var/cache/conftool/dbconfig/20221122-085832-marostegui.json |
[production] |
| 08:58 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1110.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:58 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1110.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1100 (T321126)', diff saved to https://phabricator.wikimedia.org/P40468 and previous config saved to /var/cache/conftool/dbconfig/20221122-085826-marostegui.json |
[production] |
| 08:58 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1101:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P40467 and previous config saved to /var/cache/conftool/dbconfig/20221122-085820-ladsgroup.json |
[production] |
| 08:58 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:58 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:57 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T322618)', diff saved to https://phabricator.wikimedia.org/P40466 and previous config saved to /var/cache/conftool/dbconfig/20221122-085758-ladsgroup.json |
[production] |
| 08:53 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P40465 and previous config saved to /var/cache/conftool/dbconfig/20221122-085327-ladsgroup.json |
[production] |
| 08:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T321130)', diff saved to https://phabricator.wikimedia.org/P40464 and previous config saved to /var/cache/conftool/dbconfig/20221122-084326-marostegui.json |
[production] |
| 08:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1100', diff saved to https://phabricator.wikimedia.org/P40463 and previous config saved to /var/cache/conftool/dbconfig/20221122-084320-marostegui.json |
[production] |
| 08:42 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P40462 and previous config saved to /var/cache/conftool/dbconfig/20221122-084252-ladsgroup.json |
[production] |
| 08:38 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P40461 and previous config saved to /var/cache/conftool/dbconfig/20221122-083820-ladsgroup.json |
[production] |
| 08:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1200 (T323214)', diff saved to https://phabricator.wikimedia.org/P40460 and previous config saved to /var/cache/conftool/dbconfig/20221122-083003-ladsgroup.json |
[production] |
| 08:29 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:29 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:29 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1185 (T323214)', diff saved to https://phabricator.wikimedia.org/P40459 and previous config saved to /var/cache/conftool/dbconfig/20221122-082920-ladsgroup.json |
[production] |