|
2022-03-28
ยง
|
| 10:17 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1144.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298556)', diff saved to https://phabricator.wikimedia.org/P23389 and previous config saved to /var/cache/conftool/dbconfig/20220328-101704-marostegui.json |
[production] |
| 10:13 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2033.codfw.wmnet with OS buster |
[production] |
| 10:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P23387 and previous config saved to /var/cache/conftool/dbconfig/20220328-100511-root.json |
[production] |
| 10:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P23386 and previous config saved to /var/cache/conftool/dbconfig/20220328-100159-marostegui.json |
[production] |
| 09:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P23385 and previous config saved to /var/cache/conftool/dbconfig/20220328-095007-root.json |
[production] |
| 09:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P23384 and previous config saved to /var/cache/conftool/dbconfig/20220328-094653-marostegui.json |
[production] |
| 09:46 |
<moritzm> |
installing Linux 4.9.303 on Stretch hosts |
[production] |
| 09:45 |
<mmandere@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2033.codfw.wmnet with reason: host reimage |
[production] |
| 09:43 |
<mmandere@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cp2033.codfw.wmnet with reason: host reimage |
[production] |
| 09:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1169 (re)pooling @ 10%: After schema change', diff saved to https://phabricator.wikimedia.org/P23383 and previous config saved to /var/cache/conftool/dbconfig/20220328-093503-root.json |
[production] |
| 09:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1143 (T298556)', diff saved to https://phabricator.wikimedia.org/P23382 and previous config saved to /var/cache/conftool/dbconfig/20220328-093148-marostegui.json |
[production] |
| 09:24 |
<mmandere@cumin1001> |
START - Cookbook sre.hosts.reimage for host cp2033.codfw.wmnet with OS buster |
[production] |
| 09:13 |
<moritzm> |
installing Linux 4.19.235 on Buster hosts |
[production] |
| 09:11 |
<mmandere> |
depool cp2033 for reimage - T290005 |
[production] |
| 09:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1143 (T298556)', diff saved to https://phabricator.wikimedia.org/P23379 and previous config saved to /var/cache/conftool/dbconfig/20220328-091041-marostegui.json |
[production] |
| 09:10 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1143.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:10 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1143.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298556)', diff saved to https://phabricator.wikimedia.org/P23378 and previous config saved to /var/cache/conftool/dbconfig/20220328-091033-marostegui.json |
[production] |
| 09:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1112 (re)pooling @ 100%: After downgrade ', diff saved to https://phabricator.wikimedia.org/P23377 and previous config saved to /var/cache/conftool/dbconfig/20220328-090445-root.json |
[production] |
| 09:03 |
<moritzm> |
installing Linux 5.10.106 on Bullseye hosts |
[production] |
| 08:56 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 08:55 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 08:55 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 08:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P23376 and previous config saved to /var/cache/conftool/dbconfig/20220328-085528-marostegui.json |
[production] |
| 08:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P23375 and previous config saved to /var/cache/conftool/dbconfig/20220328-085507-root.json |
[production] |
| 08:53 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 08:50 |
<jynus> |
deploy new alerting (0.7.1) for db backups at alert1001 T138562 |
[production] |
| 08:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1112 (re)pooling @ 75%: After downgrade ', diff saved to https://phabricator.wikimedia.org/P23374 and previous config saved to /var/cache/conftool/dbconfig/20220328-084941-root.json |
[production] |
| 08:47 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 08:47 |
<marostegui> |
dbmaint s1@eqiad T304812 |
[production] |
| 08:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1169 T304812', diff saved to https://phabricator.wikimedia.org/P23373 and previous config saved to /var/cache/conftool/dbconfig/20220328-084705-marostegui.json |
[production] |
| 08:46 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:773594|Enable WRITE BOTH for templatelinks normalization in more wikis (T299421)]] (duration: 00m 54s) |
[production] |
| 08:46 |
<_joe_> |
uploading conftool 2.0.0, T302471 |
[production] |
| 08:45 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 08:45 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 08:44 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 08:43 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:773938|Enable videojs in the second batch of wikis (T248418)]] (duration: 00m 55s) |
[production] |
| 08:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P23371 and previous config saved to /var/cache/conftool/dbconfig/20220328-084023-marostegui.json |
[production] |
| 08:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P23370 and previous config saved to /var/cache/conftool/dbconfig/20220328-084003-root.json |
[production] |
| 08:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1112 (re)pooling @ 50%: After downgrade ', diff saved to https://phabricator.wikimedia.org/P23369 and previous config saved to /var/cache/conftool/dbconfig/20220328-083437-root.json |
[production] |
| 08:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1142 (T298556)', diff saved to https://phabricator.wikimedia.org/P23368 and previous config saved to /var/cache/conftool/dbconfig/20220328-082518-marostegui.json |
[production] |
| 08:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P23367 and previous config saved to /var/cache/conftool/dbconfig/20220328-082459-root.json |
[production] |
| 08:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1112 (re)pooling @ 25%: After downgrade ', diff saved to https://phabricator.wikimedia.org/P23366 and previous config saved to /var/cache/conftool/dbconfig/20220328-081933-root.json |
[production] |
| 08:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1096:3316 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P23365 and previous config saved to /var/cache/conftool/dbconfig/20220328-080955-root.json |
[production] |
| 08:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1175 (re)pooling @ 100%: After downgrade', diff saved to https://phabricator.wikimedia.org/P23364 and previous config saved to /var/cache/conftool/dbconfig/20220328-080841-root.json |
[production] |
| 08:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1112 (re)pooling @ 10%: After downgrade ', diff saved to https://phabricator.wikimedia.org/P23363 and previous config saved to /var/cache/conftool/dbconfig/20220328-080429-root.json |
[production] |
| 08:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1142 (T298556)', diff saved to https://phabricator.wikimedia.org/P23362 and previous config saved to /var/cache/conftool/dbconfig/20220328-080409-marostegui.json |
[production] |
| 08:04 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1142.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:04 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1142.eqiad.wmnet with reason: Maintenance |
[production] |