101-150 of 10000 results (24ms)
2020-08-24 ยง
22:13 <rzl@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
22:10 <rzl@cumin1001> START - Cookbook sre.hosts.decommission [production]
21:29 <sbassett@deploy1001> Synchronized private/PrivateSettings.php: Deployed additional mitigations for T257687 (duration: 00m 58s) [production]
20:29 <rzl> re-enabled puppet on 'R:File = /etc/nutcracker/nutcracker.yml' T261154 [production]
19:25 <rzl> disabling puppet on 'R:File = /etc/nutcracker/nutcracker.yaml' to swap mc2028 out for mc2037 T261154 [production]
18:10 <ebernhardson@deploy1001> Synchronized wmf-config/InitialiseSettings.php: cirrus: Increase weight of grants and research namespaces in metawiki search (duration: 00m 58s) [production]
15:20 <jynus> shutdown backup2001 T260764 [production]
15:13 <hnowlan@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
15:08 <hnowlan@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
15:04 <vgutierrez> rolling restart of ats-tls to disable ECDHE-RSA-AES128-SHA - T258405 [production]
14:58 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
14:55 <rzl> switchover test complete, puppet re-enabled on cumin1001 [production]
14:54 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) [production]
14:53 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-update-tendril [production]
14:53 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) [production]
14:52 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-start-maintenance [production]
14:52 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) [production]
14:52 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-restore-ttl [production]
14:48 <rzl@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=99) [production]
14:48 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-restore-ttl [production]
14:47 <godog> powercycle ganeti5002 -- host down and nothing in console [production]
14:43 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) [production]
14:43 <rzl@cumin1001> [DRY-RUN] MediaWiki read-only period ends at: 2020-08-24 14:43:35.570234 [production]
14:43 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.07-set-readwrite [production]
14:43 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) [production]
14:43 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite [production]
14:43 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0) [production]
14:43 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions [production]
14:43 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) [production]
14:43 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki [production]
14:42 <rzl@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=99) [production]
14:42 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly [production]
14:42 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) [production]
14:41 <rzl@cumin1001> [DRY-RUN] MediaWiki read-only period starts at: 2020-08-24 14:41:55.754938 [production]
14:41 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.02-set-readonly [production]
14:41 <dcausse> creating cirrus indices for lldwiki [production]
14:39 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) [production]
14:39 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance [production]
14:38 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) [production]
14:28 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-warmup-caches [production]
14:28 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) [production]
14:28 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl [production]
14:24 <rzl@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=99) [production]
14:24 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl [production]
14:24 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) [production]
14:24 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-disable-puppet [production]
14:22 <moritzm> installing libexif security updates on stretch [production]
14:18 <rzl> disabling puppet on cumin1001 and starting a test of the DC switchover automation, expect some SAL noise but no production impact [production]
14:08 <duesen> Deployed patch for T260485 [production]
13:59 <marostegui> Stop mysql on db1117:3325 to clone db1128 - T260324 [production]