|
2024-01-25
ยง
|
| 13:27 |
<cmooney@cumin1002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2022.codfw.wmnet |
[production] |
| 13:27 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2021.codfw.wmnet |
[production] |
| 13:26 |
<cmooney@cumin1002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2021.codfw.wmnet |
[production] |
| 13:26 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2021.codfw.wmnet |
[production] |
| 13:26 |
<cmooney@cumin1002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2021.codfw.wmnet |
[production] |
| 13:25 |
<topranks> |
stopping logstash service on logstash2025 to faciliate VM migration T355549 |
[production] |
| 13:25 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2021.codfw.wmnet |
[production] |
| 13:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P55673 and previous config saved to /var/cache/conftool/dbconfig/20240125-132407-marostegui.json |
[production] |
| 13:24 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2267.codfw.wmnet with OS bullseye |
[production] |
| 13:21 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2395.codfw.wmnet with OS bullseye |
[production] |
| 13:20 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2129 (re)pooling @ 10%: After T355885', diff saved to https://phabricator.wikimedia.org/P55672 and previous config saved to /var/cache/conftool/dbconfig/20240125-132043-root.json |
[production] |
| 13:18 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2129.codfw.wmnet with reason: Maintenance |
[production] |
| 13:18 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on db2129.codfw.wmnet with reason: Maintenance |
[production] |
| 13:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2129', diff saved to https://phabricator.wikimedia.org/P55671 and previous config saved to /var/cache/conftool/dbconfig/20240125-131547-marostegui.json |
[production] |
| 13:12 |
<hashar@deploy2002> |
rebuilt and synchronized wikiversions files: group2 wikis to 1.42.0-wmf.15 refs T354433 |
[production] |
| 13:09 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P55670 and previous config saved to /var/cache/conftool/dbconfig/20240125-130900-marostegui.json |
[production] |
| 13:08 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2357.codfw.wmnet with reason: host reimage |
[production] |
| 13:05 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2267.codfw.wmnet with reason: host reimage |
[production] |
| 13:02 |
<cmooney@cumin1002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2021.codfw.wmnet |
[production] |
| 13:02 |
<topranks> |
draining VMs from ganeti2021 ahead of codfw rack b5 maintenance T355549 |
[production] |
| 13:02 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2395.codfw.wmnet with reason: host reimage |
[production] |
| 12:58 |
<hnowlan@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2267.codfw.wmnet with reason: host reimage |
[production] |
| 12:58 |
<hnowlan@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2357.codfw.wmnet with reason: host reimage |
[production] |
| 12:57 |
<hnowlan@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2395.codfw.wmnet with reason: host reimage |
[production] |
| 12:53 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2155 (T354336)', diff saved to https://phabricator.wikimedia.org/P55669 and previous config saved to /var/cache/conftool/dbconfig/20240125-125353-marostegui.json |
[production] |
| 12:41 |
<hnowlan@cumin2002> |
START - Cookbook sre.hosts.reimage for host mw2267.codfw.wmnet with OS bullseye |
[production] |
| 12:41 |
<hnowlan@cumin2002> |
START - Cookbook sre.hosts.reimage for host mw2395.codfw.wmnet with OS bullseye |
[production] |
| 12:41 |
<hnowlan@cumin2002> |
START - Cookbook sre.hosts.reimage for host mw2357.codfw.wmnet with OS bullseye |
[production] |
| 12:12 |
<jgiannelos@deploy2002> |
Finished deploy [restbase/deploy@708f0f3]: (no justification provided) (duration: 20m 28s) |
[production] |
| 12:06 |
<moritzm> |
installing openssh security updates |
[production] |
| 11:53 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2155 (T354336)', diff saved to https://phabricator.wikimedia.org/P55667 and previous config saved to /var/cache/conftool/dbconfig/20240125-115322-marostegui.json |
[production] |
| 11:53 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2187.codfw.wmnet with reason: Maintenance |
[production] |
| 11:52 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 16:00:00 on db2187.codfw.wmnet with reason: Maintenance |
[production] |
| 11:52 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2155.codfw.wmnet with reason: Maintenance |
[production] |
| 11:52 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2155.codfw.wmnet with reason: Maintenance |
[production] |
| 11:52 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2147 (T354336)', diff saved to https://phabricator.wikimedia.org/P55666 and previous config saved to /var/cache/conftool/dbconfig/20240125-115233-marostegui.json |
[production] |
| 11:52 |
<jgiannelos@deploy2002> |
Started deploy [restbase/deploy@708f0f3]: (no justification provided) |
[production] |
| 11:45 |
<zabe@deploy2002> |
Finished scap: Backport for [[gerrit:992894|Start reading from af_actor/afh_actor in group0 wikis (T355616)]] (duration: 08m 25s) |
[production] |
| 11:44 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1038.eqiad.wmnet to cluster eqiad and group D |
[production] |
| 11:42 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.addnode for new host ganeti1038.eqiad.wmnet to cluster eqiad and group D |
[production] |
| 11:38 |
<zabe@deploy2002> |
zabe: Continuing with sync |
[production] |
| 11:38 |
<zabe@deploy2002> |
zabe: Backport for [[gerrit:992894|Start reading from af_actor/afh_actor in group0 wikis (T355616)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 11:37 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P55665 and previous config saved to /var/cache/conftool/dbconfig/20240125-113727-marostegui.json |
[production] |
| 11:36 |
<zabe@deploy2002> |
Started scap: Backport for [[gerrit:992894|Start reading from af_actor/afh_actor in group0 wikis (T355616)]] |
[production] |
| 11:29 |
<hashar@deploy2002> |
Finished scap: Backport for [[gerrit:992781|UserGroupManager: Fix cross-wiki database access (T355813)]] (duration: 08m 50s) |
[production] |
| 11:26 |
<claime> |
Restarting ferm.service on k8s node kubernetes2036.codfw.wmnet - T354855 |
[production] |
| 11:26 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2107.codfw.wmnet with reason: Maintenance |
[production] |
| 11:26 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2107.codfw.wmnet with reason: Maintenance |
[production] |
| 11:23 |
<hashar@deploy2002> |
hashar and zabe: Continuing with sync |
[production] |
| 11:22 |
<hashar@deploy2002> |
hashar and zabe: Backport for [[gerrit:992781|UserGroupManager: Fix cross-wiki database access (T355813)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |