3601-3650 of 10000 results (38ms)
2021-02-17 ยง
15:13 <root@cumin1001> START - Cookbook sre.hosts.reboot-single for host dumpsdata1001.eqiad.wmnet [production]
14:26 <cdanis> starting rolling restart of cp-upload@eqsin varnish-fe T275028 [production]
13:55 <marostegui@cumin1001> dbctl commit (dc=all): 'db1172 (re)pooling @ 100%: Slowly repool db1172', diff saved to https://phabricator.wikimedia.org/P14396 and previous config saved to /var/cache/conftool/dbconfig/20210217-135533-root.json [production]
13:40 <marostegui@cumin1001> dbctl commit (dc=all): 'db1172 (re)pooling @ 80%: Slowly repool db1172', diff saved to https://phabricator.wikimedia.org/P14395 and previous config saved to /var/cache/conftool/dbconfig/20210217-134030-root.json [production]
13:30 <akosiaris@deploy1001> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
13:30 <akosiaris@deploy1001> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
13:28 <moritzm> installing libzstd security updates on Buster [production]
13:25 <marostegui@cumin1001> dbctl commit (dc=all): 'db1172 (re)pooling @ 60%: Slowly repool db1172', diff saved to https://phabricator.wikimedia.org/P14393 and previous config saved to /var/cache/conftool/dbconfig/20210217-132526-root.json [production]
13:19 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:664593|Enable Wikibase Repo ID generator rate limiting on Wikidata (T272032)]] (duration: 01m 11s) [production]
13:10 <marostegui@cumin1001> dbctl commit (dc=all): 'db1172 (re)pooling @ 50%: Slowly repool db1172', diff saved to https://phabricator.wikimedia.org/P14392 and previous config saved to /var/cache/conftool/dbconfig/20210217-131022-root.json [production]
13:06 <akosiaris@deploy1001> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
13:05 <akosiaris@deploy1001> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
12:55 <akosiaris@deploy1001> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
12:55 <akosiaris@deploy1001> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
12:55 <marostegui@cumin1001> dbctl commit (dc=all): 'db1172 (re)pooling @ 40%: Slowly repool db1172', diff saved to https://phabricator.wikimedia.org/P14391 and previous config saved to /var/cache/conftool/dbconfig/20210217-125519-root.json [production]
12:50 <akosiaris@deploy1001> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
12:49 <akosiaris@deploy1001> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
12:45 <akosiaris@deploy1001> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
12:45 <akosiaris@deploy1001> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
12:42 <akosiaris@deploy1001> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
12:42 <akosiaris@deploy1001> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
12:40 <akosiaris@deploy1001> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
12:40 <marostegui@cumin1001> dbctl commit (dc=all): 'db1172 (re)pooling @ 20%: Slowly repool db1172', diff saved to https://phabricator.wikimedia.org/P14390 and previous config saved to /var/cache/conftool/dbconfig/20210217-124015-root.json [production]
12:40 <akosiaris@deploy1001> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
12:14 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 6eeee95e090408c8bd35d14c2f76e3afd8a59048: vector: Enable search treatment AB test on test wikis (T259798) (duration: 01m 08s) [production]
12:10 <urbanecm@deploy1001> Synchronized dblists/desktop-improvements.dblist: 7872251778b65cb03eb5457f1b901d208d514609: Revert "Revert "vector: Enable WVUI search on test wikis"" (T259798) (duration: 01m 09s) [production]
12:07 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 7872251778b65cb03eb5457f1b901d208d514609: Revert "Revert "vector: Enable WVUI search on test wikis"" (T259798) (duration: 01m 25s) [production]
11:58 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2001.wikimedia.org [production]
11:55 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single for host netbox-dev2001.wikimedia.org [production]
11:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly pool db1172 in s8 - T258361', diff saved to https://phabricator.wikimedia.org/P14389 and previous config saved to /var/cache/conftool/dbconfig/20210217-112422-marostegui.json [production]
11:08 <akosiaris@deploy1001> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
11:08 <akosiaris@deploy1001> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
11:04 <akosiaris@deploy1001> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
11:04 <akosiaris@deploy1001> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
11:04 <akosiaris@deploy1001> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
11:03 <akosiaris@deploy1001> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
10:52 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on cloudnet1004.eqiad.wmnet with reason: hardware failure [production]
10:52 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on cloudnet1004.eqiad.wmnet with reason: hardware failure [production]
10:13 <_joe_> depooling mw1331 to perform some tests for T266855 [production]
10:08 <aborrero@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
10:01 <aborrero@cumin1001> START - Cookbook sre.dns.netbox [production]
09:32 <elukey> reboot dbstore100[3-5] for kernel upgrades [production]
08:44 <marostegui> upgrade es2020 es2021 es2022's kernel [production]
08:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly pool db1172 in s8 - T258361', diff saved to https://phabricator.wikimedia.org/P14388 and previous config saved to /var/cache/conftool/dbconfig/20210217-084120-marostegui.json [production]
08:08 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1004.eqiad.wmnet [production]
08:04 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host stat1004.eqiad.wmnet [production]
07:41 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly pool db1172 in s8 - T258361', diff saved to https://phabricator.wikimedia.org/P14387 and previous config saved to /var/cache/conftool/dbconfig/20210217-074107-marostegui.json [production]
07:40 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1007.eqiad.wmnet [production]
07:33 <elukey@cumin1001> START - Cookbook sre.hosts.reboot-single for host stat1007.eqiad.wmnet [production]
07:30 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1006.eqiad.wmnet [production]