|
2022-11-14
ยง
|
| 10:16 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1143.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321130)', diff saved to https://phabricator.wikimedia.org/P39406 and previous config saved to /var/cache/conftool/dbconfig/20221114-101637-marostegui.json |
[production] |
| 10:12 |
<vgutierrez> |
upgrading acme-chief on acmechief1001 to version 0.35 (requires disabling puppet on R:acme_chief::cert) |
[production] |
| 10:09 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39405 and previous config saved to /var/cache/conftool/dbconfig/20221114-100931-ladsgroup.json |
[production] |
| 10:09 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:855743|Rework SpecialPagesWithoutScans query (T322849)]] (duration: 11m 17s) |
[production] |
| 10:07 |
<vgutierrez> |
upload acme-chief 0.35 to apt.wm.o (buster-wikimedia) - T244232 T262251 |
[production] |
| 10:07 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1105:3312 (T322618)', diff saved to https://phabricator.wikimedia.org/P39404 and previous config saved to /var/cache/conftool/dbconfig/20221114-100720-ladsgroup.json |
[production] |
| 10:07 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:07 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:06 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39403 and previous config saved to /var/cache/conftool/dbconfig/20221114-100650-root.json |
[production] |
| 10:06 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:05 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2104 (T322618)', diff saved to https://phabricator.wikimedia.org/P39402 and previous config saved to /var/cache/conftool/dbconfig/20221114-100515-ladsgroup.json |
[production] |
| 10:02 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2104 (T322618)', diff saved to https://phabricator.wikimedia.org/P39401 and previous config saved to /var/cache/conftool/dbconfig/20221114-100254-ladsgroup.json |
[production] |
| 10:02 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance |
[production] |
| 10:02 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2104.codfw.wmnet with reason: Maintenance |
[production] |
| 10:02 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance |
[production] |
| 10:02 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance |
[production] |
| 10:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P39400 and previous config saved to /var/cache/conftool/dbconfig/20221114-100131-marostegui.json |
[production] |
| 09:58 |
<ladsgroup@deploy1002> |
ladsgroup and ladsgroup: Backport for [[gerrit:855743|Rework SpecialPagesWithoutScans query (T322849)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 09:57 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:855743|Rework SpecialPagesWithoutScans query (T322849)]] |
[production] |
| 09:57 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance |
[production] |
| 09:56 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance |
[production] |
| 09:56 |
<ladsgroup@cumin1001> |
END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 6:00:00 on db2100.codfw.wmnet with reason: Maintenance |
[production] |
| 09:56 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2100.codfw.wmnet with reason: Maintenance |
[production] |
| 09:56 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance |
[production] |
| 09:56 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance |
[production] |
| 09:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39399 and previous config saved to /var/cache/conftool/dbconfig/20221114-095145-root.json |
[production] |
| 09:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P39398 and previous config saved to /var/cache/conftool/dbconfig/20221114-094624-marostegui.json |
[production] |
| 09:44 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.addnode for new host ganeti1033.eqiad.wmnet to cluster eqiad and group D |
[production] |
| 09:39 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788 |
[production] |
| 09:38 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 4788 |
[production] |
| 09:38 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 50083 |
[production] |
| 09:38 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'email' for AS: 50083 |
[production] |
| 09:37 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 32934 |
[production] |
| 09:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39397 and previous config saved to /var/cache/conftool/dbconfig/20221114-093640-root.json |
[production] |
| 09:35 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 32934 |
[production] |
| 09:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142 (T321130)', diff saved to https://phabricator.wikimedia.org/P39396 and previous config saved to /var/cache/conftool/dbconfig/20221114-093118-marostegui.json |
[production] |
| 09:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39395 and previous config saved to /var/cache/conftool/dbconfig/20221114-092135-root.json |
[production] |
| 09:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1142 (T321130)', diff saved to https://phabricator.wikimedia.org/P39394 and previous config saved to /var/cache/conftool/dbconfig/20221114-091934-marostegui.json |
[production] |
| 09:19 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1142.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:19 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 5:00:00 on db1142.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141 (T321130)', diff saved to https://phabricator.wikimedia.org/P39393 and previous config saved to /var/cache/conftool/dbconfig/20221114-091912-marostegui.json |
[production] |
| 09:18 |
<ayounsi@cumin1001> |
END (ERROR) - Cookbook sre.network.peering (exit_code=97) with action 'configure' for AS: 13335 |
[production] |
| 09:18 |
<ayounsi@cumin1001> |
START - Cookbook sre.network.peering with action 'configure' for AS: 13335 |
[production] |
| 09:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39392 and previous config saved to /var/cache/conftool/dbconfig/20221114-090630-root.json |
[production] |
| 09:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P39391 and previous config saved to /var/cache/conftool/dbconfig/20221114-090406-marostegui.json |
[production] |
| 08:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 3%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39389 and previous config saved to /var/cache/conftool/dbconfig/20221114-085125-root.json |
[production] |
| 08:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P39388 and previous config saved to /var/cache/conftool/dbconfig/20221114-084859-marostegui.json |
[production] |
| 08:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2145 (re)pooling @ 1%: After upgrade', diff saved to https://phabricator.wikimedia.org/P39387 and previous config saved to /var/cache/conftool/dbconfig/20221114-083620-root.json |
[production] |