401-450 of 10000 results (24ms)
2020-06-30 §
13:32 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:31 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:30 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:07 <hashar@deploy1001> Started scap: testwikis wikis to 1.35.0-wmf.39 [production]
12:03 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:03 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:35 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:35 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:33 <awight> EU BACON cooked [production]
11:32 <awight@deploy1001> Synchronized wmf-config/InitialiseSettings.php: BACON: [[gerrit:608478|Configure TeWü survey on dewiki (take 2) (T253112)]] (duration: 00m 58s) [production]
11:32 <jayme> restarted docker-reporter-base-images and docker-reporter-releng-images on deneb - T253396 [production]
11:31 <jayme> pushed a scratch docker image as docker-registry.discovery.wmnet/envoy-tls-local-proxy:dontuseme - T253396 [production]
11:28 <awight@deploy1001> Synchronized php-1.35.0-wmf.38/extensions/QuickSurveys: BACON: [[gerrit:608477|Embedded surveys are hidden when no element is available (T256627)]] (duration: 00m 56s) [production]
11:26 <awight@deploy1001> Synchronized php-1.35.0-wmf.38/extensions/FileImporter: BACON: [[gerrit:608476|Set Status error if permission check returns false. (T256428)]] (duration: 00m 58s) [production]
11:13 <ema> deneb: systemctl restart docker-reporter-base-images.service [production]
10:59 <ema> upload librdkafka 0.11.6-1.1wmf1 to buster-wikimedia https://phabricator.wikimedia.org/P11703 T256444 [production]
10:59 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:59 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1076', diff saved to https://phabricator.wikimedia.org/P11710 and previous config saved to /var/cache/conftool/dbconfig/20200630-105254-marostegui.json [production]
10:45 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:45 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:41 <ema> cp2040: restart purged and varnishkafka to use updated librdkafka1 T256444 [production]
10:38 <ema> cp2040: upgrade librdkafka1 to 0.11.6-1.1wmf1 https://phabricator.wikimedia.org/P11703 T256444 [production]
10:37 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:37 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:30 <hashar@deploy1001> Synchronized php-1.35.0-wmf.39/includes/specials/SpecialUndelete.php: Remove another use of PageArchive::getRevision - T249982 T254176 (duration: 00m 56s) [production]
10:09 <marostegui> Deploy schema change on db1076 [production]
10:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1076', diff saved to https://phabricator.wikimedia.org/P11708 and previous config saved to /var/cache/conftool/dbconfig/20200630-100912-marostegui.json [production]
10:04 <vgutierrez> rolling restart of eqiad cache nodes to catch up on kernel upgrades [production]
10:03 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:03 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:02 <volker-e@deploy1001> Finished deploy [design/style-guide@e3fda83]: Deploy design/style-guide: (duration: 00m 07s) [production]
10:02 <volker-e@deploy1001> Started deploy [design/style-guide@e3fda83]: Deploy design/style-guide: [production]
09:47 <hashar@deploy1001> Pruned MediaWiki: 1.35.0-wmf.37 (duration: 02m 20s) [production]
09:40 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:40 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:21 <hashar@deploy1001> Pruned MediaWiki: 1.35.0-wmf.36 (duration: 28m 11s) [production]
08:54 <vgutierrez@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:53 <vgutierrez@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:53 <hashar@deploy1001> clean aborted: Pruned MediaWiki: 1.35.0-wmf.36 (duration: 00m 00s) [production]
08:51 <hashar> Applied security patches to wmf/1.35.0-wmf.39 # T254176 [production]
08:51 <vgutierrez> rolling restart of codfw cp nodes after "re-formatting" nvme devices - T256655 [production]
08:23 <vgutierrez> repool cp3053 - T256632 [production]
08:10 <hashar> 1.35.0-wmf.39 was branched at e169e3dabcb2217809fc41ba44b43a39ae1a678e T254176 [production]
08:05 <marostegui> Stop MySQL on db1117:3322 to clone db1080 (this will trigger haproxy alerts) - T256717 [production]
08:05 <vgutierrez> powercycle cp3053 (unresponsive after reboot) - T256632 [production]
08:01 <jbond42> disable puppet to restart puppetmasters front ends [production]
07:42 <vgutierrez> reboot cp3053 - T256632 [production]
05:51 <jhuneidi@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . [production]
05:13 <marostegui> Deploy schema change on s8 codfw - T256680 [production]