251-300 of 10000 results (23ms)
2020-11-24 ยง
15:42 <marostegui> Drop kraken user from s4 - T268636 [production]
15:38 <elukey> move druid1005 from rack B7 to B6 - T267065 [production]
15:35 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
15:33 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
15:29 <otto@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . [production]
15:29 <otto@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
15:28 <jayme> pushed docker-registry.discovery.wmnet/calico/kube-controllers:v3.17.0 docker-registry.discovery.wmnet/calico/node:v3.17.0 docker-registry.discovery.wmnet/calico/typha:v3.17.0 [production]
15:23 <jayme> imported calico 3.17.0 into component/calico-future for stretch-wikimedia [production]
15:07 <godog> swift eqiad-prod: decom ms-be1022 ssd from swift - T267870 [production]
15:01 <marostegui> Enable GTID on clouddb1013:3311 clouddb1015:3314 clouddb1017:3311 clouddb1019:3314 T267090 [production]
14:58 <elukey> move analytics1072 from rack B2 to B3 - T267065 [production]
14:54 <otto@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . [production]
14:54 <otto@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
14:53 <jayme> imported helmfile 0.135.0-1 into buster-wikimedia and stretch-wikimedia [production]
14:47 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
14:44 <akosiaris@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
14:43 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
14:43 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
14:42 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
14:42 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1085 for schema change', diff saved to https://phabricator.wikimedia.org/P13392 and previous config saved to /var/cache/conftool/dbconfig/20201124-144219-marostegui.json [production]
14:34 <liw> finished testing Scap on Beta cluster in prep for https://phabricator.wikimedia.org/T268634 [production]
14:31 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
14:27 <otto@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
14:19 <marostegui@cumin1001> dbctl commit (dc=all): 'db1121 (re)pooling @ 100%: After cloning the new clouddb hosts', diff saved to https://phabricator.wikimedia.org/P13391 and previous config saved to /var/cache/conftool/dbconfig/20201124-141912-root.json [production]
14:09 <moritzm> reset-failed idp-u2f.service after Hiera change (one time issue, will soon be obsolete) [production]
14:04 <marostegui@cumin1001> dbctl commit (dc=all): 'db1121 (re)pooling @ 75%: After cloning the new clouddb hosts', diff saved to https://phabricator.wikimedia.org/P13390 and previous config saved to /var/cache/conftool/dbconfig/20201124-140409-root.json [production]
13:52 <elukey@deploy1001> Finished deploy [statsv/statsv@b25b6ff]: Deploy https://gerrit.wikimedia.org/r/c/analytics/statsv/+/643252 (duration: 00m 05s) [production]
13:52 <elukey@deploy1001> Started deploy [statsv/statsv@b25b6ff]: Deploy https://gerrit.wikimedia.org/r/c/analytics/statsv/+/643252 [production]
13:49 <marostegui@cumin1001> dbctl commit (dc=all): 'db1121 (re)pooling @ 50%: After cloning the new clouddb hosts', diff saved to https://phabricator.wikimedia.org/P13389 and previous config saved to /var/cache/conftool/dbconfig/20201124-134905-root.json [production]
13:40 <marostegui> Stop MySQL on db1074 to clone clouddb1018 and clouddb1014 T267090 [production]
13:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1074 to clone clouddb1018 and clouddb1014 T267090', diff saved to https://phabricator.wikimedia.org/P13388 and previous config saved to /var/cache/conftool/dbconfig/20201124-133709-marostegui.json [production]
13:34 <marostegui@cumin1001> dbctl commit (dc=all): 'db1121 (re)pooling @ 25%: After cloning the new clouddb hosts', diff saved to https://phabricator.wikimedia.org/P13387 and previous config saved to /var/cache/conftool/dbconfig/20201124-133402-root.json [production]
13:13 <jgleeson> civicrm revision is 28464df973, config revision is 928918a9b6 [production]
13:01 <hashar@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.18 [production]
13:01 <liw> done testing Scap release candidate on beta (failed: disk full on deploy01) [production]
12:49 <hnowlan> disabled cassandra service on restbase2009, starting drain [production]
12:30 <liw> testing upcoming Scap release on beta [production]
12:02 <ayounsi@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
11:59 <jayme> imported helm3 3.4.1-1 into buster-wikimedia and stretch-wikimedia [production]
11:56 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
11:52 <XioNoX> push CR641949 and CR641949 [production]
11:38 <effie> rolling depool and pool app and api clusters - T244340 [production]
11:25 <_joe_> rebuild docker images for T268612 [production]
11:20 <effie> disable puppet on api and app servers to rollout onhost memcached - T244340 [production]
11:15 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:15 <root@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:15 <root@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:14 <root@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:12 <marostegui> Stop mysql on db1125:3312 to clone clouddb1014:3312 and clouddb1018:3312 - T267090 [production]
10:45 <moritzm> upgrading seaborgium to Buster [production]