2301-2350 of 10000 results (36ms)
2020-06-08 ยง
23:23 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
23:02 <ryankemper@cumin2001> END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99) [production]
22:58 <ryankemper@cumin2001> START - Cookbook sre.elasticsearch.rolling-upgrade [production]
22:53 <ryankemper@cumin2001> END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99) [production]
22:53 <shdubsh> update mtail to 3.0.0~rc35 on mw and wtp hosts codfw [production]
22:49 <eileen> civicrm revision changed from 11b0e7c7e5 to be4c5a4951, config revision is 386b9bc457 [production]
22:49 <ryankemper@cumin2001> START - Cookbook sre.elasticsearch.rolling-upgrade [production]
20:52 <Amir1> applying the sql alter table on [[gerrit:594292|ipblocks]] on labswiki (T251188) [production]
20:27 <RoanKattouw> Running initUserPreference.php -s growthexperiments-homepage-enable -t growthexperiments-help-panel-tog-help-panel on wikis that have GrowthExperiments installed (T240920) [production]
18:56 <Urbanecm> Morning <del>SWAT</del>config/backport window done [production]
18:56 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 1630a10: Set wgProofreadPagePageJoiner to __PAGEJOIN__ for zhwikisource (T205826) (duration: 00m 58s) [production]
18:55 <urbanecm@deploy1001> sync-file aborted: SWAT: 1630a10: Set wgProofreadPagePageJoiner to __PAGEJOIN__ for zhwikisource (duration: 00m 00s) [production]
18:51 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 0e85203: Enable subpages in Page namespace on napwikisource (T252755) (duration: 00m 58s) [production]
18:44 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
18:42 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
18:28 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: End GrowthExperiments homepage A/B test (T254413) (duration: 00m 57s) [production]
18:23 <catrope@deploy1001> Synchronized wmf-config/CommonSettings.php: Disable HTCP purges for testwiki (T250781) (part 2) (duration: 00m 56s) [production]
18:20 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Disable HTCP purges for testwiki (T250781) (part 1) (duration: 00m 59s) [production]
17:50 <elukey> restart prometheus burrow exporter for kafka main on kafkamon1001 - T254498 [production]
17:43 <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.35/resources/src/mediawiki.misc-authed-curate/rollback.js: Fix: Diff pages show rollback confirmation prompt if there is the "Mark as patrolled" link (T254538) (duration: 00m 59s) [production]
17:14 <elukey@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-mirror-maker (exit_code=0) [production]
16:55 <elukey@cumin1001> START - Cookbook sre.kafka.roll-restart-mirror-maker [production]
16:54 <elukey@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) [production]
16:44 <liw> testing upcoming Scap release on beta [production]
15:29 <hnowlan> Migrated all cpjobqueue jobs from scb to Kubernetes [production]
15:29 <hnowlan@deploy1001> Finished deploy [cpjobqueue/deploy@07d8c32]: Disabling jobs migrated to k8s (duration: 04m 34s) [production]
15:28 <jynus@cumin2001> dbctl commit (dc=all): 'depool db2075 for mw maintenance T254139', diff saved to https://phabricator.wikimedia.org/P11411 and previous config saved to /var/cache/conftool/dbconfig/20200608-152811-jynus.json [production]
15:24 <hnowlan@deploy1001> Started deploy [cpjobqueue/deploy@07d8c32]: Disabling jobs migrated to k8s [production]
15:12 <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.35/extensions/Wikibase/client/includes/Store/Sql/DirectSqlStore.php: Wrap WAN-cached PropertyInfoLookup with an APCu cache, Part III out of III (T254536) (duration: 00m 57s) [production]
15:10 <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.35/extensions/Wikibase/repo/includes/Store/Sql/SqlStore.php: Wrap WAN-cached PropertyInfoLookup with an APCu cache, Part II out of III (T254536) (duration: 00m 57s) [production]
15:09 <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.35/extensions/Wikibase/lib/includes/Store/CachingPropertyInfoLookup.php: Wrap WAN-cached PropertyInfoLookup with an APCu cache, Part I out of III (T254536) (duration: 00m 59s) [production]
15:05 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'production' . [production]
14:53 <cdanis> โœ”๏ธ cdanis@cumin1001.eqiad.wmnet ~ ๐Ÿ•šโ˜• sudo cumin A:mw-canary 'enable-puppet "cdanis deploying I25ab44c1 T252605"' [production]
14:52 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' . [production]
14:48 <papaul> powering down ms-be2016 for BBU replacement [production]
14:47 <cdanis> โœ”๏ธ cdanis@cumin1001.eqiad.wmnet ~ ๐Ÿ•šโ˜• sudo cumin A:mw-canary 'disable-puppet "cdanis deploying I25ab44c1 T252605"' [production]
14:41 <moritzm> upgrading mw API servers in codfw to PHP 7.2.31 [production]
14:00 <jbond42> updating puppet-merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/602738/4 [production]
13:58 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:58 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
13:50 <urbanecm@deploy1001> Synchronized private/PrivateSettings.php: Update mitigations for T250887 (duration: 00m 57s) [production]
13:41 <elukey@cumin1001> START - Cookbook sre.kafka.roll-restart-brokers [production]
12:23 <XioNoX> repool codfw - T243080 [production]
12:18 <XioNoX> rollback cr2-codfw vrrp/ospf/bgp changes - T243080 [production]
12:18 <marostegui> Compress InnoDB on db2094:3311 T254462 [production]
12:09 <XioNoX> cr2-codfw> request chassis routing-engine master switch - T243080 [production]
12:05 <XioNoX> reboot cr2-codfw:re0 (backup) - T243080 [production]
11:53 <XioNoX> cr2-codfw> request chassis routing-engine master switch - T243080 [production]
11:53 <moritzm> restarting dnsdist on malmok [production]
11:53 <marostegui> Deploy schema change on s3 - T251188 [production]