751-800 of 10000 results (38ms)
2020-08-24 ยง
14:58 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
14:55 <rzl> switchover test complete, puppet re-enabled on cumin1001 [production]
14:54 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) [production]
14:53 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-update-tendril [production]
14:53 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) [production]
14:52 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-start-maintenance [production]
14:52 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) [production]
14:52 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-restore-ttl [production]
14:48 <rzl@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=99) [production]
14:48 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-restore-ttl [production]
14:47 <godog> powercycle ganeti5002 -- host down and nothing in console [production]
14:43 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) [production]
14:43 <rzl@cumin1001> [DRY-RUN] MediaWiki read-only period ends at: 2020-08-24 14:43:35.570234 [production]
14:43 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.07-set-readwrite [production]
14:43 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) [production]
14:43 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite [production]
14:43 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0) [production]
14:43 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions [production]
14:43 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) [production]
14:43 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki [production]
14:42 <rzl@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=99) [production]
14:42 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly [production]
14:42 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) [production]
14:41 <rzl@cumin1001> [DRY-RUN] MediaWiki read-only period starts at: 2020-08-24 14:41:55.754938 [production]
14:41 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.02-set-readonly [production]
14:41 <dcausse> creating cirrus indices for lldwiki [production]
14:39 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) [production]
14:39 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance [production]
14:38 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) [production]
14:28 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-warmup-caches [production]
14:28 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) [production]
14:28 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl [production]
14:24 <rzl@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=99) [production]
14:24 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl [production]
14:24 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) [production]
14:24 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-disable-puppet [production]
14:22 <moritzm> installing libexif security updates on stretch [production]
14:18 <rzl> disabling puppet on cumin1001 and starting a test of the DC switchover automation, expect some SAL noise but no production impact [production]
14:08 <duesen> Deployed patch for T260485 [production]
13:59 <marostegui> Stop mysql on db1117:3325 to clone db1128 - T260324 [production]
13:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1092 for MCR change', diff saved to https://phabricator.wikimedia.org/P12327 and previous config saved to /var/cache/conftool/dbconfig/20200824-135538-marostegui.json [production]
13:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1101:3318 after MCR change', diff saved to https://phabricator.wikimedia.org/P12326 and previous config saved to /var/cache/conftool/dbconfig/20200824-133032-marostegui.json [production]
13:27 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:25 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1101:3318', diff saved to https://phabricator.wikimedia.org/P12325 and previous config saved to /var/cache/conftool/dbconfig/20200824-131305-marostegui.json [production]
13:05 <moritzm> installing imagemagick security updates on stretch [production]
13:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1101:3318', diff saved to https://phabricator.wikimedia.org/P12323 and previous config saved to /var/cache/conftool/dbconfig/20200824-130024-marostegui.json [production]
12:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1101:3318', diff saved to https://phabricator.wikimedia.org/P12322 and previous config saved to /var/cache/conftool/dbconfig/20200824-125131-marostegui.json [production]
12:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1084 for MCR change', diff saved to https://phabricator.wikimedia.org/P12321 and previous config saved to /var/cache/conftool/dbconfig/20200824-122848-marostegui.json [production]
12:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1105:3311 after MCR change', diff saved to https://phabricator.wikimedia.org/P12320 and previous config saved to /var/cache/conftool/dbconfig/20200824-122752-marostegui.json [production]