7301-7350 of 10000 results (43ms)
2021-06-25 §
07:58 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1003.eqiad.wmnet [production]
07:57 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-etcd1003.eqiad.wmnet [production]
07:42 <moritzm> imported Jenkins 2.289.1 to thirdparty/ci for buster-wikimedia T285531 [production]
07:30 <dcausse> depool and restart blazegraph on wdqs1005 [production]
07:17 <dcausse> installing openjdk-8-dbg on wdqs1005 to debug blazegraph [production]
2021-06-24 §
23:02 <legoktm> reverted cumin1001 spicerack live hacks [production]
22:57 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) [production]
22:55 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-start-maintenance [production]
22:55 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) [production]
22:55 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-restore-ttl [production]
22:36 <volans> set x2 codfw master back to RW [production]
22:30 <legoktm@cumin1001> END (ERROR) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=97) [production]
22:29 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly [production]
22:29 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) [production]
22:29 <legoktm@cumin1001> [DRY-RUN] MediaWiki read-only period starts at: 2021-06-24 22:29:25.643909 [production]
22:29 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.02-set-readonly [production]
22:09 <legoktm@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=99) [production]
22:09 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance [production]
22:06 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) [production]
22:05 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-warmup-caches [production]
22:04 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) [production]
22:04 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl [production]
22:01 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) [production]
22:01 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-disable-puppet [production]
21:59 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) [production]
21:58 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-disable-puppet [production]
21:47 <legoktm> live hacked spicerack on cumin1001 to revert https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/700963/ [production]
20:58 <legoktm> starting dry run and live test of DC switchover [production]
20:53 <legoktm> legoktm@phab1001:~$ sudo /srv/phab/phabricator/bin/remove destroy M320 (spam) [production]
20:44 <volans> uploaded spicerack_0.0.55 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia [production]
20:28 <legoktm> re-enabled daily digests for wikimedia-l - T285486 [production]
19:10 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.37.0-wmf.11 [production]
19:07 <dduvall@deploy1002> Synchronized php: group1 wikis to 1.37.0-wmf.11 (duration: 01m 04s) [production]
19:06 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.11 [production]
19:04 <dduvall> preparing to roll group2 to 1.37.0-wmf.11 (T281152) (cc risky patch contacts Amir1 Krinkle DannyS712) [production]
19:02 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:56 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
18:21 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:18 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
18:17 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:12 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
17:12 <dduvall@deploy1002> Synchronized php: group1 wikis to 1.37.0-wmf.11 (duration: 01m 06s) [production]
17:11 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.11 [production]
17:08 <dduvall> re-rolling group1 to 1.37.0-wmf.11 (T281152) following deployment of blocker fixes (cc risky patch contacts Amir1 Krinkle DannyS712) [production]
16:12 <twentyafterfour> restarted php7.3-fpm on phab1001 [production]
15:43 <hnowlan> running `nodetool decommission` on maps2007 [production]
15:42 <hnowlan@puppetmaster1001> conftool action : set/pooled=no; selector: name=maps2007.codfw.wmnet [production]
15:42 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on maps2007.codfw.wmnet with reason: depooling and reimaging as buster replica [production]
15:42 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on maps2007.codfw.wmnet with reason: depooling and reimaging as buster replica [production]
15:31 <moritzm> installing jackson-databind security updates [production]