3201-3250 of 10000 results (29ms)
2020-08-25 §
00:56 <cdanis> T259621 ❌cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 homer 'cr*' commit 'drain cr2-esams transport link' [production]
00:36 <cdanis> T259621 cdanis@re1.cr3-esams> request chassis routing-engine master switch [production]
00:30 <cdanis> T259621 cdanis@re1.cr3-esams> request vmhost reboot re0 [production]
00:24 <cdanis> T259621 cdanis@re1.cr3-esams> request vmhost software add /var/tmp/junos-vmhost-install-mx-x86-64-17.3R3-S8.1.tgz re0 [production]
00:18 <cdanis> T259621 cdanis@re0.cr3-esams> request chassis routing-engine master switch [production]
00:14 <cdanis> T259621 cdanis@re0.cr3-esams> request vmhost reboot re1 [production]
00:08 <cdanis> T259621 cdanis@re0.cr3-esams> request vmhost software add /var/tmp/junos-vmhost-install-mx-x86-64-17.3R3-S8.1.tgz re1 [production]
2020-08-24 §
23:46 <cdanis> depool esams T259621 [production]
23:16 <Urbanecm> Evening B&C window done [production]
23:06 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: 778f710bbbdb24730f7ce4c75d5ff1ca7a5ce3b3: Alternate configuration mechanism for Parsoid (T241961) (duration: 00m 58s) [production]
22:13 <rzl@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
22:10 <rzl@cumin1001> START - Cookbook sre.hosts.decommission [production]
21:29 <sbassett@deploy1001> Synchronized private/PrivateSettings.php: Deployed additional mitigations for T257687 (duration: 00m 58s) [production]
20:29 <rzl> re-enabled puppet on 'R:File = /etc/nutcracker/nutcracker.yml' T261154 [production]
19:25 <rzl> disabling puppet on 'R:File = /etc/nutcracker/nutcracker.yaml' to swap mc2028 out for mc2037 T261154 [production]
18:10 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: cirrus: Increase weight of grants and research namespaces in metawiki search (duration: 00m 58s) [production]
15:20 <jynus> shutdown backup2001 T260764 [production]
15:13 <hnowlan@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
15:08 <hnowlan@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
15:04 <vgutierrez> rolling restart of ats-tls to disable ECDHE-RSA-AES128-SHA - T258405 [production]
14:58 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
14:55 <rzl> switchover test complete, puppet re-enabled on cumin1001 [production]
14:54 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) [production]
14:53 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-update-tendril [production]
14:53 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) [production]
14:52 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-start-maintenance [production]
14:52 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) [production]
14:52 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-restore-ttl [production]
14:48 <rzl@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=99) [production]
14:48 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-restore-ttl [production]
14:47 <godog> powercycle ganeti5002 -- host down and nothing in console [production]
14:43 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) [production]
14:43 <rzl@cumin1001> [DRY-RUN] MediaWiki read-only period ends at: 2020-08-24 14:43:35.570234 [production]
14:43 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.07-set-readwrite [production]
14:43 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) [production]
14:43 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite [production]
14:43 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0) [production]
14:43 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions [production]
14:43 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) [production]
14:43 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki [production]
14:42 <rzl@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=99) [production]
14:42 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly [production]
14:42 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) [production]
14:41 <rzl@cumin1001> [DRY-RUN] MediaWiki read-only period starts at: 2020-08-24 14:41:55.754938 [production]
14:41 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.02-set-readonly [production]
14:41 <dcausse> creating cirrus indices for lldwiki [production]
14:39 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) [production]
14:39 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance [production]
14:38 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) [production]
14:28 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-warmup-caches [production]