|
2021-10-19
ยง
|
| 11:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1101:3317 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17522 and previous config saved to /var/cache/conftool/dbconfig/20211019-110009-root.json |
[production] |
| 10:56 |
<marostegui> |
Upgrade clouddb1021 |
[production] |
| 10:53 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2025.codfw.wmnet |
[production] |
| 10:51 |
<moritzm> |
failover master in ganeti-test to ganeti2026 |
[production] |
| 10:50 |
<godog> |
bounce superset on an-tool1005 to pick up statsd changes - T247963 |
[production] |
| 10:49 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2112.codfw.wmnet with OS stretch |
[production] |
| 10:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1101:3318 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17521 and previous config saved to /var/cache/conftool/dbconfig/20211019-104829-root.json |
[production] |
| 10:45 |
<godog> |
bounce navtiming on webperf1001 to pick up statsd changes - T247963 |
[production] |
| 10:45 |
<godog> |
bounce superset on an-tool1010 to pick up statsd changes - T247963 |
[production] |
| 10:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1101:3317 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17520 and previous config saved to /var/cache/conftool/dbconfig/20211019-104506-root.json |
[production] |
| 10:38 |
<oblivian@deploy1002> |
Synchronized w/static.php: Config: [[gerrit:730182|static.php: Add support for /static/current rewrites (take 2) (T285232)]] (duration: 00m 55s) |
[production] |
| 10:38 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2026.codfw.wmnet |
[production] |
| 10:37 |
<marostegui> |
Upgrade db1101 (s7,s8) |
[production] |
| 10:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1101 (s7,s8) for upgrade', diff saved to https://phabricator.wikimedia.org/P17519 and previous config saved to /var/cache/conftool/dbconfig/20211019-103634-marostegui.json |
[production] |
| 10:34 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 10:31 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 10:29 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 10:28 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 10:27 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2026.codfw.wmnet |
[production] |
| 10:22 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 10:22 |
<oblivian@deploy1002> |
Synchronized tests/WmfConfigServicesTest.php: Config: [[gerrit:731918|ProductionServices: use graphite2003 for statsd (T247963)]] (duration: 00m 54s) |
[production] |
| 10:22 |
<godog> |
flip mw statsd traffic with https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/731918 - T247963 |
[production] |
| 10:21 |
<oblivian@deploy1002> |
Synchronized wmf-config/ProductionServices.php: Config: [[gerrit:731918|ProductionServices: use graphite2003 for statsd (T247963)]] (duration: 00m 54s) |
[production] |
| 10:20 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 10:18 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db2112.codfw.wmnet with OS stretch |
[production] |
| 10:16 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db2112.codfw.wmnet with OS buster |
[production] |
| 09:52 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db2112.codfw.wmnet with OS buster |
[production] |
| 09:50 |
<marostegui@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db2112.codfw.wmnet with OS buster |
[production] |
| 09:44 |
<hashar@deploy1002> |
Pruned MediaWiki: 1.38.0-wmf.3 (duration: 01m 39s) |
[production] |
| 09:42 |
<hashar@deploy1002> |
Pruned MediaWiki: 1.38.0-wmf.2 (duration: 16m 06s) |
[production] |
| 09:37 |
<godog> |
move graphite/statsd writes to graphite2003 - T247963 |
[production] |
| 09:34 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host db2112.codfw.wmnet with OS buster |
[production] |
| 09:27 |
<hashar> |
sap clean --delete 1.38.0-wmf.2 && scap clean --delete 1.38.0-wmf.3 # T281169 |
[production] |
| 09:27 |
<hashar> |
Cloned and applied security patches for 1.38.0-wmf.5 # T281169 |
[production] |
| 09:19 |
<marostegui> |
Stop slave on db2112 T290865 |
[production] |
| 09:18 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 14 hosts with reason: Schema change s1 T281058 |
[production] |
| 09:18 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on 14 hosts with reason: Schema change s1 T281058 |
[production] |
| 09:03 |
<XioNoX> |
push anycast tuning to all Telia transit links - T288843 |
[production] |
| 08:50 |
<godog> |
point graphite.discovery.wmnet to graphite2003 - T247963 |
[production] |
| 08:40 |
<XioNoX> |
push prep-work for anycast tuning to all sites - T288843 |
[production] |
| 08:33 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 13 hosts with reason: Schema change s8 T281058 |
[production] |
| 08:33 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on 13 hosts with reason: Schema change s8 T281058 |
[production] |
| 08:32 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript namespaceDupes.php hrwiki --fix |
[production] |
| 08:17 |
<oblivian@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 08:07 |
<mvernon@cumin2002> |
conftool action : set/pooled=false; selector: name=codfw,dnsdisc=swift |
[production] |
| 08:07 |
<mvernon@cumin2002> |
conftool action : set/pooled=false; selector: name=codfw,dnsdisc=swift-ro |
[production] |
| 08:03 |
<XioNoX> |
push prep-work for anycast tuning in ulsfo (try 2) - T288843 |
[production] |
| 08:01 |
<oblivian@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 07:32 |
<oblivian@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 07:24 |
<ema> |
A:cp start rolling varnish upgrades to 6.0.8-1wm1 T292290 |
[production] |