|
2021-10-19
ยง
|
| 11:55 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1101:3318 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17531 and previous config saved to /var/cache/conftool/dbconfig/20211019-114844-root.json |
[production] |
| 11:46 |
<marostegui> |
Upgrade db1105 (s1,s2) |
[production] |
| 11:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1105 (s1,s2) for upgrade', diff saved to https://phabricator.wikimedia.org/P17530 and previous config saved to /var/cache/conftool/dbconfig/20211019-114649-marostegui.json |
[production] |
| 11:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1101:3317 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17529 and previous config saved to /var/cache/conftool/dbconfig/20211019-114520-root.json |
[production] |
| 11:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1101:3318 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17527 and previous config saved to /var/cache/conftool/dbconfig/20211019-113340-root.json |
[production] |
| 11:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1101:3317 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17526 and previous config saved to /var/cache/conftool/dbconfig/20211019-113017-root.json |
[production] |
| 11:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1101:3318 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17525 and previous config saved to /var/cache/conftool/dbconfig/20211019-111837-root.json |
[production] |
| 11:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1101:3317 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17524 and previous config saved to /var/cache/conftool/dbconfig/20211019-111513-root.json |
[production] |
| 11:11 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:09 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:08 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 7c31b04e50101a60db7ae8acae64bc031f5e1007: DPL: Explicitly note it is not possible to enable DPL on any more wikis (duration: 00m 55s) |
[production] |
| 11:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1101:3318 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17523 and previous config saved to /var/cache/conftool/dbconfig/20211019-110333-root.json |
[production] |
| 11:02 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2025.codfw.wmnet |
[production] |
| 11:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1101:3317 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17522 and previous config saved to /var/cache/conftool/dbconfig/20211019-110009-root.json |
[production] |
| 10:56 |
<marostegui> |
Upgrade clouddb1021 |
[production] |
| 10:53 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2025.codfw.wmnet |
[production] |
| 10:51 |
<moritzm> |
failover master in ganeti-test to ganeti2026 |
[production] |
| 10:50 |
<godog> |
bounce superset on an-tool1005 to pick up statsd changes - T247963 |
[production] |
| 10:49 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2112.codfw.wmnet with OS stretch |
[production] |
| 10:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1101:3318 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17521 and previous config saved to /var/cache/conftool/dbconfig/20211019-104829-root.json |
[production] |
| 10:45 |
<godog> |
bounce navtiming on webperf1001 to pick up statsd changes - T247963 |
[production] |
| 10:45 |
<godog> |
bounce superset on an-tool1010 to pick up statsd changes - T247963 |
[production] |
| 10:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1101:3317 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17520 and previous config saved to /var/cache/conftool/dbconfig/20211019-104506-root.json |
[production] |
| 10:38 |
<oblivian@deploy1002> |
Synchronized w/static.php: Config: [[gerrit:730182|static.php: Add support for /static/current rewrites (take 2) (T285232)]] (duration: 00m 55s) |
[production] |
| 10:38 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2026.codfw.wmnet |
[production] |
| 10:37 |
<marostegui> |
Upgrade db1101 (s7,s8) |
[production] |
| 10:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101 (s7,s8) for upgrade', diff saved to https://phabricator.wikimedia.org/P17519 and previous config saved to /var/cache/conftool/dbconfig/20211019-103634-marostegui.json |
[production] |
| 10:34 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 10:31 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 10:29 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 10:28 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 10:27 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2026.codfw.wmnet |
[production] |
| 10:22 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 10:22 |
<oblivian@deploy1002> |
Synchronized tests/WmfConfigServicesTest.php: Config: [[gerrit:731918|ProductionServices: use graphite2003 for statsd (T247963)]] (duration: 00m 54s) |
[production] |
| 10:22 |
<godog> |
flip mw statsd traffic with https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/731918 - T247963 |
[production] |
| 10:21 |
<oblivian@deploy1002> |
Synchronized wmf-config/ProductionServices.php: Config: [[gerrit:731918|ProductionServices: use graphite2003 for statsd (T247963)]] (duration: 00m 54s) |
[production] |
| 10:20 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 10:18 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db2112.codfw.wmnet with OS stretch |
[production] |
| 10:16 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db2112.codfw.wmnet with OS buster |
[production] |
| 09:52 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db2112.codfw.wmnet with OS buster |
[production] |
| 09:50 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db2112.codfw.wmnet with OS buster |
[production] |
| 09:44 |
<hashar@deploy1002> |
Pruned MediaWiki: 1.38.0-wmf.3 (duration: 01m 39s) |
[production] |
| 09:42 |
<hashar@deploy1002> |
Pruned MediaWiki: 1.38.0-wmf.2 (duration: 16m 06s) |
[production] |
| 09:37 |
<godog> |
move graphite/statsd writes to graphite2003 - T247963 |
[production] |
| 09:34 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db2112.codfw.wmnet with OS buster |
[production] |
| 09:27 |
<hashar> |
sap clean --delete 1.38.0-wmf.2 && scap clean --delete 1.38.0-wmf.3 # T281169 |
[production] |
| 09:27 |
<hashar> |
Cloned and applied security patches for 1.38.0-wmf.5 # T281169 |
[production] |
| 09:19 |
<marostegui> |
Stop slave on db2112 T290865 |
[production] |
| 09:18 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 14 hosts with reason: Schema change s1 T281058 |
[production] |