8001-8050 of 10000 results (100ms)
2023-02-22 ยง
14:06 <TheresNoTime> UTC afternoon backport window not done due to in-progress deployment [production]
14:06 <bking@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
12:40 <moritzm> rolling restart of FPM on mw canaries [production]
12:32 <moritzm> installing openssl security updates on buster [production]
12:22 <akosiaris> test [production]
12:18 <cgoubert@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) [production]
12:18 <cgoubert@cumin1001> MediaWiki read-only period ends at: 2023-02-22 12:18:45.829060 [production]
12:18 <cgoubert@cumin1001> START - Cookbook sre.switchdc.mediawiki.07-set-readwrite [production]
12:18 <cgoubert@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) [production]
12:18 <cgoubert@cumin1001> MediaWiki read-only period ends at: 2023-02-22 12:18:11.451680 [production]
12:18 <cgoubert@cumin1001> START - Cookbook sre.switchdc.mediawiki.07-set-readwrite [production]
12:16 <akosiaris@cumin1001> conftool action : set/val=false; selector: name=ReadOnly [production]
12:14 <cgoubert@cumin1001> conftool action : set/val=false; selector: name=ReadOnly,scope=codfw [production]
12:13 <cgoubert@cumin1001> conftool action : set/val=no; selector: name=ReadOnly,scope=codfw [production]
12:13 <cgoubert@cumin1001> conftool action : set/val=False; selector: name=ReadOnly,scope=codfw [production]
12:02 <moritzm> installing NSS security updates [production]
11:57 <cgoubert@cumin1001> conftool action : set/val=false; selector: name=ReadOnly,scope=codfw [production]
11:49 <jynus@cumin1001> dbctl commit (dc=all): 'Repool db1166, errors not fixed', diff saved to https://phabricator.wikimedia.org/P44727 and previous config saved to /var/cache/conftool/dbconfig/20230222-114940-jynus.json [production]
11:45 <jynus@cumin1001> dbctl commit (dc=all): 'Depool db1166, seen mw errors', diff saved to https://phabricator.wikimedia.org/P44726 and previous config saved to /var/cache/conftool/dbconfig/20230222-114515-jynus.json [production]
11:26 <moritzm> installing git security updates [production]
11:24 <cgoubert@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters (exit_code=0) [production]
11:18 <cgoubert@cumin1001> START - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters [production]
11:17 <cgoubert@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.09-restore-ttl (exit_code=0) [production]
11:16 <cgoubert@cumin1001> START - Cookbook sre.switchdc.mediawiki.09-restore-ttl [production]
11:16 <cgoubert@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) [production]
11:15 <cgoubert@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-start-maintenance [production]
11:14 <cgoubert@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-restart-envoy-on-jobrunners (exit_code=0) [production]
11:14 <cgoubert@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-restart-envoy-on-jobrunners [production]
11:13 <cgoubert@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) [production]
11:13 <cgoubert@cumin1001> [DRY-RUN] MediaWiki read-only period ends at: 2023-02-22 11:13:51.466468 [production]
11:13 <cgoubert@cumin1001> START - Cookbook sre.switchdc.mediawiki.07-set-readwrite [production]
11:13 <cgoubert@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) [production]
11:13 <cgoubert@cumin1001> START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite [production]
11:13 <eoghan@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gitlab2002.wikimedia.org with reason: Running failover to gitlab1003 - T329930 [production]
11:13 <cgoubert@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) [production]
11:13 <cgoubert@cumin1001> START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki [production]
11:13 <eoghan@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on gitlab2002.wikimedia.org with reason: Running failover to gitlab1003 - T329930 [production]
11:04 <cgoubert@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=99) [production]
11:03 <cgoubert@cumin1001> START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly [production]
11:03 <cgoubert@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) [production]
11:03 <cgoubert@cumin1001> [DRY-RUN] MediaWiki read-only period starts at: 2023-02-22 11:03:19.149671 [production]
11:03 <cgoubert@cumin1001> START - Cookbook sre.switchdc.mediawiki.02-set-readonly [production]
11:02 <cgoubert@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) [production]
11:02 <cgoubert@cumin1001> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance [production]
11:01 <cgoubert@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks (exit_code=0) [production]
11:01 <cgoubert@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks [production]
11:01 <cgoubert@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) [production]
11:01 <cgoubert@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-disable-puppet [production]
10:47 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2019.codfw.wmnet with OS bullseye [production]
10:45 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2021.codfw.wmnet with OS bullseye [production]