1201-1250 of 10000 results (23ms)
2021-10-19 ยง
12:03 <marostegui@cumin1001> dbctl commit (dc=all): 'db1101:3318 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17534 and previous config saved to /var/cache/conftool/dbconfig/20211019-120348-root.json [production]
12:01 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.5/extensions/WikibaseMediaInfo/: ec0125770775c1a1a54c3b592d86d287fd9e3ad6: Escape captions when writing stored data into js state (T293556) (duration: 00m 55s) [production]
12:01 <marostegui@cumin1001> dbctl commit (dc=all): 'db1105:3312 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17533 and previous config saved to /var/cache/conftool/dbconfig/20211019-120132-root.json [production]
12:00 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.4/extensions/WikibaseMediaInfo/: 79808a90a95dd5dac2b532b87fb7ec1a490ea0f0: Escape captions when writing stored data into js state (T293556) (duration: 00m 56s) [production]
12:00 <marostegui@cumin1001> dbctl commit (dc=all): 'db1101:3317 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17532 and previous config saved to /var/cache/conftool/dbconfig/20211019-120024-root.json [production]
11:58 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:56 <XioNoX> push anycast tuning to Tele2, Init7, DT transit links - T288843 [production]
11:55 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:48 <marostegui@cumin1001> dbctl commit (dc=all): 'db1101:3318 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17531 and previous config saved to /var/cache/conftool/dbconfig/20211019-114844-root.json [production]
11:46 <marostegui> Upgrade db1105 (s1,s2) [production]
11:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1105 (s1,s2) for upgrade', diff saved to https://phabricator.wikimedia.org/P17530 and previous config saved to /var/cache/conftool/dbconfig/20211019-114649-marostegui.json [production]
11:45 <marostegui@cumin1001> dbctl commit (dc=all): 'db1101:3317 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17529 and previous config saved to /var/cache/conftool/dbconfig/20211019-114520-root.json [production]
11:33 <marostegui@cumin1001> dbctl commit (dc=all): 'db1101:3318 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17527 and previous config saved to /var/cache/conftool/dbconfig/20211019-113340-root.json [production]
11:30 <marostegui@cumin1001> dbctl commit (dc=all): 'db1101:3317 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17526 and previous config saved to /var/cache/conftool/dbconfig/20211019-113017-root.json [production]
11:18 <marostegui@cumin1001> dbctl commit (dc=all): 'db1101:3318 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17525 and previous config saved to /var/cache/conftool/dbconfig/20211019-111837-root.json [production]
11:15 <marostegui@cumin1001> dbctl commit (dc=all): 'db1101:3317 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17524 and previous config saved to /var/cache/conftool/dbconfig/20211019-111513-root.json [production]
11:11 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:09 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:08 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 7c31b04e50101a60db7ae8acae64bc031f5e1007: DPL: Explicitly note it is not possible to enable DPL on any more wikis (duration: 00m 55s) [production]
11:03 <marostegui@cumin1001> dbctl commit (dc=all): 'db1101:3318 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17523 and previous config saved to /var/cache/conftool/dbconfig/20211019-110333-root.json [production]
11:02 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2025.codfw.wmnet [production]
11:00 <marostegui@cumin1001> dbctl commit (dc=all): 'db1101:3317 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17522 and previous config saved to /var/cache/conftool/dbconfig/20211019-110009-root.json [production]
10:56 <marostegui> Upgrade clouddb1021 [production]
10:53 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2025.codfw.wmnet [production]
10:51 <moritzm> failover master in ganeti-test to ganeti2026 [production]
10:50 <godog> bounce superset on an-tool1005 to pick up statsd changes - T247963 [production]
10:49 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2112.codfw.wmnet with OS stretch [production]
10:48 <marostegui@cumin1001> dbctl commit (dc=all): 'db1101:3318 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17521 and previous config saved to /var/cache/conftool/dbconfig/20211019-104829-root.json [production]
10:45 <godog> bounce navtiming on webperf1001 to pick up statsd changes - T247963 [production]
10:45 <godog> bounce superset on an-tool1010 to pick up statsd changes - T247963 [production]
10:45 <marostegui@cumin1001> dbctl commit (dc=all): 'db1101:3317 (re)pooling @ 5%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17520 and previous config saved to /var/cache/conftool/dbconfig/20211019-104506-root.json [production]
10:38 <oblivian@deploy1002> Synchronized w/static.php: Config: [[gerrit:730182|static.php: Add support for /static/current rewrites (take 2) (T285232)]] (duration: 00m 55s) [production]
10:38 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2026.codfw.wmnet [production]
10:37 <marostegui> Upgrade db1101 (s7,s8) [production]
10:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1101 (s7,s8) for upgrade', diff saved to https://phabricator.wikimedia.org/P17519 and previous config saved to /var/cache/conftool/dbconfig/20211019-103634-marostegui.json [production]
10:34 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:29 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
10:28 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
10:27 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2026.codfw.wmnet [production]
10:22 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:22 <oblivian@deploy1002> Synchronized tests/WmfConfigServicesTest.php: Config: [[gerrit:731918|ProductionServices: use graphite2003 for statsd (T247963)]] (duration: 00m 54s) [production]
10:22 <godog> flip mw statsd traffic with https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/731918 - T247963 [production]
10:21 <oblivian@deploy1002> Synchronized wmf-config/ProductionServices.php: Config: [[gerrit:731918|ProductionServices: use graphite2003 for statsd (T247963)]] (duration: 00m 54s) [production]
10:20 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:18 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db2112.codfw.wmnet with OS stretch [production]
10:16 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db2112.codfw.wmnet with OS buster [production]
09:52 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host db2112.codfw.wmnet with OS buster [production]
09:50 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db2112.codfw.wmnet with OS buster [production]
09:44 <hashar@deploy1002> Pruned MediaWiki: 1.38.0-wmf.3 (duration: 01m 39s) [production]