2301-2350 of 10000 results (90ms)
2024-01-25 ยง
13:18 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2129.codfw.wmnet with reason: Maintenance [production]
13:18 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2129.codfw.wmnet with reason: Maintenance [production]
13:15 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2129', diff saved to https://phabricator.wikimedia.org/P55671 and previous config saved to /var/cache/conftool/dbconfig/20240125-131547-marostegui.json [production]
13:12 <hashar@deploy2002> rebuilt and synchronized wikiversions files: group2 wikis to 1.42.0-wmf.15 refs T354433 [production]
13:09 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P55670 and previous config saved to /var/cache/conftool/dbconfig/20240125-130900-marostegui.json [production]
13:08 <hnowlan@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2357.codfw.wmnet with reason: host reimage [production]
13:05 <hnowlan@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2267.codfw.wmnet with reason: host reimage [production]
13:02 <cmooney@cumin1002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2021.codfw.wmnet [production]
13:02 <topranks> draining VMs from ganeti2021 ahead of codfw rack b5 maintenance T355549 [production]
13:02 <hnowlan@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2395.codfw.wmnet with reason: host reimage [production]
12:58 <hnowlan@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2267.codfw.wmnet with reason: host reimage [production]
12:58 <hnowlan@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2357.codfw.wmnet with reason: host reimage [production]
12:57 <hnowlan@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw2395.codfw.wmnet with reason: host reimage [production]
12:53 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2155 (T354336)', diff saved to https://phabricator.wikimedia.org/P55669 and previous config saved to /var/cache/conftool/dbconfig/20240125-125353-marostegui.json [production]
12:41 <hnowlan@cumin2002> START - Cookbook sre.hosts.reimage for host mw2267.codfw.wmnet with OS bullseye [production]
12:41 <hnowlan@cumin2002> START - Cookbook sre.hosts.reimage for host mw2395.codfw.wmnet with OS bullseye [production]
12:41 <hnowlan@cumin2002> START - Cookbook sre.hosts.reimage for host mw2357.codfw.wmnet with OS bullseye [production]
12:12 <jgiannelos@deploy2002> Finished deploy [restbase/deploy@708f0f3]: (no justification provided) (duration: 20m 28s) [production]
12:06 <moritzm> installing openssh security updates [production]
11:53 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db2155 (T354336)', diff saved to https://phabricator.wikimedia.org/P55667 and previous config saved to /var/cache/conftool/dbconfig/20240125-115322-marostegui.json [production]
11:53 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
11:52 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 16:00:00 on db2187.codfw.wmnet with reason: Maintenance [production]
11:52 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
11:52 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db2155.codfw.wmnet with reason: Maintenance [production]
11:52 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2147 (T354336)', diff saved to https://phabricator.wikimedia.org/P55666 and previous config saved to /var/cache/conftool/dbconfig/20240125-115233-marostegui.json [production]
11:52 <jgiannelos@deploy2002> Started deploy [restbase/deploy@708f0f3]: (no justification provided) [production]
11:45 <zabe@deploy2002> Finished scap: Backport for [[gerrit:992894|Start reading from af_actor/afh_actor in group0 wikis (T355616)]] (duration: 08m 25s) [production]
11:44 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti1038.eqiad.wmnet to cluster eqiad and group D [production]
11:42 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti1038.eqiad.wmnet to cluster eqiad and group D [production]
11:38 <zabe@deploy2002> zabe: Continuing with sync [production]
11:38 <zabe@deploy2002> zabe: Backport for [[gerrit:992894|Start reading from af_actor/afh_actor in group0 wikis (T355616)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
11:37 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P55665 and previous config saved to /var/cache/conftool/dbconfig/20240125-113727-marostegui.json [production]
11:36 <zabe@deploy2002> Started scap: Backport for [[gerrit:992894|Start reading from af_actor/afh_actor in group0 wikis (T355616)]] [production]
11:29 <hashar@deploy2002> Finished scap: Backport for [[gerrit:992781|UserGroupManager: Fix cross-wiki database access (T355813)]] (duration: 08m 50s) [production]
11:26 <claime> Restarting ferm.service on k8s node kubernetes2036.codfw.wmnet - T354855 [production]
11:26 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2107.codfw.wmnet with reason: Maintenance [production]
11:26 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2107.codfw.wmnet with reason: Maintenance [production]
11:23 <hashar@deploy2002> hashar and zabe: Continuing with sync [production]
11:22 <hashar@deploy2002> hashar and zabe: Backport for [[gerrit:992781|UserGroupManager: Fix cross-wiki database access (T355813)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
11:22 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P55664 and previous config saved to /var/cache/conftool/dbconfig/20240125-112220-marostegui.json [production]
11:20 <hashar@deploy2002> Started scap: Backport for [[gerrit:992781|UserGroupManager: Fix cross-wiki database access (T355813)]] [production]
11:07 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2147 (T354336)', diff saved to https://phabricator.wikimedia.org/P55663 and previous config saved to /var/cache/conftool/dbconfig/20240125-110714-marostegui.json [production]
11:05 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2147.codfw.wmnet with reason: Maintenance [production]
11:05 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db2147.codfw.wmnet with reason: Maintenance [production]
11:05 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance [production]
11:05 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on db2139.codfw.wmnet with reason: Maintenance [production]
11:05 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2138:3314 (T354336)', diff saved to https://phabricator.wikimedia.org/P55662 and previous config saved to /var/cache/conftool/dbconfig/20240125-110521-marostegui.json [production]
10:57 <taavi@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudrabbit1003.eqiad.wmnet with OS bookworm [production]
10:50 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2138:3314', diff saved to https://phabricator.wikimedia.org/P55660 and previous config saved to /var/cache/conftool/dbconfig/20240125-105015-marostegui.json [production]
10:39 <taavi@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudrabbit1003.eqiad.wmnet with reason: host reimage [production]