1901-1950 of 10000 results (31ms)
2020-06-12 §
10:36 <jmm@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
10:33 <moritzm> rolling restart of the ulsfo ganeti cluster [production]
10:21 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) [production]
10:02 <filippo@cumin1001> START - Cookbook sre.hosts.upgrade-and-reboot [production]
10:01 <jmm@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:01 <filippo@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) [production]
10:01 <filippo@cumin1001> START - Cookbook sre.hosts.reboot-single [production]
10:01 <jmm@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Include db2084 in dbctl, depooled', diff saved to https://phabricator.wikimedia.org/P11480 and previous config saved to /var/cache/conftool/dbconfig/20200612-095855-marostegui.json [production]
09:58 <godog> roll-restart thanos-fe / thanos-be for microcode updates [production]
08:51 <elukey> restart gerrit on gerrit1001 [production]
08:48 <elukey> update cr1/cr2 analyitics filters for T252767 and T252675 [production]
08:44 <marostegui> Compress InnoDB on db2092 - T254462 [production]
08:36 <marostegui> Clone db2084 from db2080 [production]
08:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2080 to clone db2084', diff saved to https://phabricator.wikimedia.org/P11478 and previous config saved to /var/cache/conftool/dbconfig/20200612-083231-marostegui.json [production]
08:24 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:22 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db2084 from s4 and s5', diff saved to https://phabricator.wikimedia.org/P11477 and previous config saved to /var/cache/conftool/dbconfig/20200612-081455-marostegui.json [production]
07:56 <elukey> depool mw1384 [production]
07:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2084 from s4 and s5', diff saved to https://phabricator.wikimedia.org/P11476 and previous config saved to /var/cache/conftool/dbconfig/20200612-075202-marostegui.json [production]
07:26 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:24 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:08 <marostegui> Reimage db2086 [production]
07:07 <elukey> depool/scap pull/pool mw1384 [production]
07:05 <moritzm> installing intel-microcode security updates (regressions have been sorted out) [production]
05:42 <moritzm> installing stretch kernel security updates (no reboots yet) [production]
05:40 <moritzm> installing buster kernel security updates (no reboots yet) [production]
04:54 <marostegui> Deploy schema change on s6 codfw - T250066 [production]
01:02 <ejegg> updated payments-wiki from aceddff8b5 to 5fd4eb1519 [production]
00:10 <Amir1> BACON is done [production]
2020-06-11 §
23:54 <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.36/extensions/Wikibase: [[gerrit:604845|Fix entity id lookup for interwiki special page links (T255078)]] (duration: 00m 38s) [production]
23:51 <ladsgroup@deploy1001> scap failed: average error rate on 3/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/e474f13ffac6b8c3bf919c4aeafc8c9b for details) [production]
23:43 <ladsgroup@deploy1001> Synchronized wmf-config/extension-list: [[gerrit:604778|Remove ContributionTracking extension]] (T255216), Part III (duration: 00m 57s) [production]
23:42 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:604778|Remove ContributionTracking extension]] (T255216), Part II (duration: 00m 58s) [production]
23:38 <ladsgroup@deploy1001> Synchronized wmf-config/CommonSettings.php: [[gerrit:604778|Remove ContributionTracking extension]] (T255216), Part I (duration: 00m 59s) [production]
23:37 <Reedy> create cn_notice_regions on metawiki and testwiki T252596 [production]
20:34 <pt1979@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:31 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
20:15 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
20:13 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
20:00 <pt1979@cumin2001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
19:59 <jhuneidi@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.36 [production]
19:58 <pt1979@cumin2001> START - Cookbook sre.hosts.downtime [production]
19:33 <akosiaris> apply emergency sessionstore fixes in codfw as well [production]
19:32 <akosiaris@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'sessionstore' for release 'production' . [production]
19:20 <gilles@deploy1001> Finished deploy [performance/asoranking@0a096c4]: T252424 (duration: 00m 47s) [production]
19:19 <gilles@deploy1001> Started deploy [performance/asoranking@0a096c4]: T252424 [production]
19:12 <akosiaris> repool eqiad for sessionstore [production]
19:12 <akosiaris@cumin1001> conftool action : set/pooled=true; selector: name=eqiad,dnsdisc=sessionstore [production]
19:10 <akosiaris> remove the podaffinity restrictions for sessionstore in eqiad [production]