3851-3900 of 10000 results (31ms)
2020-08-19 ยง
15:26 <rzl@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=99) [production]
15:26 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl [production]
15:22 <rzl@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=99) [production]
15:22 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-restore-ttl [production]
15:18 <godog> prometheus codfw lvextend --resizefs --size +80G /dev/mapper/vg--ssd-prometheus--ops [production]
15:17 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) [production]
15:17 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl [production]
15:16 <rzl@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=99) [production]
15:16 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl [production]
15:14 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) [production]
15:14 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-restore-ttl [production]
15:08 <rzl@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=99) [production]
15:08 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-restore-ttl [production]
15:06 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
15:04 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
14:50 <rzl@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=99) [production]
14:50 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl [production]
14:50 <rzl> running the switchdc cookbooks with --live-test, simulating a switch to eqiad where we're already running, no production impact is expected [production]
14:47 <rzl@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) [production]
14:47 <rzl@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-disable-puppet [production]
14:41 <rzl> disable puppet on cumin1001 for switchdc testing [production]
14:35 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
14:33 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
14:27 <ppchelko@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
13:38 <ppchelko@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
13:34 <gehel> depooling wdqs1007 and restarting blazegraph [production]
13:29 <_joe_> depooling and disabling puppet on restbase1024 for further investigation [production]
13:27 <ppchelko@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
13:26 <ppchelko@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
13:25 <ppchelko@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
13:10 <pt1979@cumin2001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:03 <_joe_> building and uploading fluent-bit, ratelimit images [production]
13:01 <pt1979@cumin2001> START - Cookbook sre.dns.netbox [production]
12:57 <_joe_> building a new version of the base docker images [production]
11:29 <awight> EU bacon finished [production]
11:28 <effie> restart mwdebug* servers [production]
11:08 <awight@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:621227|Fix typos in flaggedrevs comments ()]] (duration: 01m 19s) [production]
09:22 <oblivian@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) [production]
08:48 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
08:48 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
08:43 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
08:43 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
08:36 <XioNoX> update firewall policies on pfw - T260585 [production]
08:35 <jayme> running puppet on A:all-mw-eqiad [production]
08:23 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
08:23 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) [production]
08:20 <godog> switch grafana.w.o to grafana 7 in codfw - T259143 [production]
08:19 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
08:18 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
08:14 <oblivian@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]