|
2024-01-23
ยง
|
| 12:56 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host snapshot1016.eqiad.wmnet with OS bullseye |
[production] |
| 12:54 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2168:3318', diff saved to https://phabricator.wikimedia.org/P55362 and previous config saved to /var/cache/conftool/dbconfig/20240123-125459-marostegui.json |
[production] |
| 12:49 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1173 (re)pooling @ 50%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55361 and previous config saved to /var/cache/conftool/dbconfig/20240123-124859-root.json |
[production] |
| 12:45 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host snapshot1017.eqiad.wmnet |
[production] |
| 12:39 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2168:3318', diff saved to https://phabricator.wikimedia.org/P55360 and previous config saved to /var/cache/conftool/dbconfig/20240123-123952-marostegui.json |
[production] |
| 12:33 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1173 (re)pooling @ 25%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55359 and previous config saved to /var/cache/conftool/dbconfig/20240123-123354-root.json |
[production] |
| 12:33 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2171:3316 (re)pooling @ 100%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55358 and previous config saved to /var/cache/conftool/dbconfig/20240123-123346-root.json |
[production] |
| 12:31 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.migrate-host for host snapshot1017.eqiad.wmnet |
[production] |
| 12:28 |
<claime> |
Restarting killed maintenance job mediawiki_job_MachineVision_prioritize_uncategorized.service |
[production] |
| 12:26 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for sretest1001.eqiad.wmnet |
[production] |
| 12:26 |
<kamila@cumin1002> |
START - Cookbook sre.hosts.remove-downtime for sretest1001.eqiad.wmnet |
[production] |
| 12:26 |
<kamila@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest1001.eqiad.wmnet with reason: testing the cookbook |
[production] |
| 12:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2168:3318 (T354336)', diff saved to https://phabricator.wikimedia.org/P55357 and previous config saved to /var/cache/conftool/dbconfig/20240123-122446-marostegui.json |
[production] |
| 12:23 |
<kamila@cumin1002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on sretest1001.eqiad.wmnet with reason: testing the cookbook |
[production] |
| 12:23 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2168:3318 (T354336)', diff saved to https://phabricator.wikimedia.org/P55356 and previous config saved to /var/cache/conftool/dbconfig/20240123-122336-marostegui.json |
[production] |
| 12:23 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2168.codfw.wmnet with reason: Maintenance |
[production] |
| 12:23 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2168.codfw.wmnet with reason: Maintenance |
[production] |
| 12:23 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3318 (T354336)', diff saved to https://phabricator.wikimedia.org/P55355 and previous config saved to /var/cache/conftool/dbconfig/20240123-122314-marostegui.json |
[production] |
| 12:21 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2171:3315 (re)pooling @ 100%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55354 and previous config saved to /var/cache/conftool/dbconfig/20240123-122105-root.json |
[production] |
| 12:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1173 (re)pooling @ 10%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55353 and previous config saved to /var/cache/conftool/dbconfig/20240123-121849-root.json |
[production] |
| 12:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2171:3316 (re)pooling @ 75%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55352 and previous config saved to /var/cache/conftool/dbconfig/20240123-121841-root.json |
[production] |
| 12:17 |
<claime> |
Restarting ferm.service on k8s node mw1495.eqiad.wmnet - T354855 |
[production] |
| 12:16 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host snapshot1016.eqiad.wmnet |
[production] |
| 12:14 |
<claime> |
scap::dsh::scap_proxies: Replace mw1486 by mw1405 - T355622 |
[production] |
| 12:13 |
<Amir1> |
dropping bv2015_edits table from all wikis (T355594) |
[production] |
| 12:08 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3318', diff saved to https://phabricator.wikimedia.org/P55351 and previous config saved to /var/cache/conftool/dbconfig/20240123-120807-marostegui.json |
[production] |
| 12:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2171:3315 (re)pooling @ 75%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55350 and previous config saved to /var/cache/conftool/dbconfig/20240123-120600-root.json |
[production] |
| 12:05 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.migrate-host for host snapshot1016.eqiad.wmnet |
[production] |
| 12:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1173 (re)pooling @ 5%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55349 and previous config saved to /var/cache/conftool/dbconfig/20240123-120344-root.json |
[production] |
| 12:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2171:3316 (re)pooling @ 50%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55348 and previous config saved to /var/cache/conftool/dbconfig/20240123-120335-root.json |
[production] |
| 12:02 |
<Amir1> |
dropping bv2009_edits table from all wikis (T355594) |
[production] |
| 12:02 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host snapshot1017.eqiad.wmnet with OS bullseye |
[production] |
| 11:54 |
<godog> |
initial cleanup of replicated thanos blocks - T351927 |
[production] |
| 11:53 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3318', diff saved to https://phabricator.wikimedia.org/P55347 and previous config saved to /var/cache/conftool/dbconfig/20240123-115301-marostegui.json |
[production] |
| 11:50 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2171:3315 (re)pooling @ 50%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55346 and previous config saved to /var/cache/conftool/dbconfig/20240123-115055-root.json |
[production] |
| 11:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1173 (re)pooling @ 1%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55345 and previous config saved to /var/cache/conftool/dbconfig/20240123-114840-root.json |
[production] |
| 11:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2171:3316 (re)pooling @ 25%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55344 and previous config saved to /var/cache/conftool/dbconfig/20240123-114831-root.json |
[production] |
| 11:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1173', diff saved to https://phabricator.wikimedia.org/P55343 and previous config saved to /var/cache/conftool/dbconfig/20240123-114826-marostegui.json |
[production] |
| 11:37 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2167:3318 (T354336)', diff saved to https://phabricator.wikimedia.org/P55342 and previous config saved to /var/cache/conftool/dbconfig/20240123-113754-marostegui.json |
[production] |
| 11:35 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2171:3315 (re)pooling @ 25%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55341 and previous config saved to /var/cache/conftool/dbconfig/20240123-113550-root.json |
[production] |
| 11:35 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2167:3318 (T354336)', diff saved to https://phabricator.wikimedia.org/P55340 and previous config saved to /var/cache/conftool/dbconfig/20240123-113544-marostegui.json |
[production] |
| 11:35 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2167.codfw.wmnet with reason: Maintenance |
[production] |
| 11:35 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db2167.codfw.wmnet with reason: Maintenance |
[production] |
| 11:35 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2166 (T354336)', diff saved to https://phabricator.wikimedia.org/P55339 and previous config saved to /var/cache/conftool/dbconfig/20240123-113522-marostegui.json |
[production] |
| 11:35 |
<marostegui> |
Starting s6 eqiad failover from db1173 to db1231 - T355660 |
[production] |
| 11:34 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on snapshot1017.eqiad.wmnet with reason: host reimage |
[production] |
| 11:31 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on snapshot1017.eqiad.wmnet with reason: host reimage |
[production] |
| 11:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2171:3316 (re)pooling @ 10%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55338 and previous config saved to /var/cache/conftool/dbconfig/20240123-112420-root.json |
[production] |
| 11:22 |
<Amir1> |
dropping bv2011_edits table from all wikis (T355594) |
[production] |
| 11:20 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2171:3315 (re)pooling @ 10%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55337 and previous config saved to /var/cache/conftool/dbconfig/20240123-112045-root.json |
[production] |