4101-4150 of 10000 results (74ms)
2019-07-30 §
16:33 <elukey@cumin1001> START - Cookbook sre.kafka.roll-restart-brokers [production]
16:22 <godog> bounce rsyslog on centrallog1001 - T199406 [production]
15:41 <elukey@cumin1001> END (FAIL) - Cookbook sre.kafka.roll-restart-brokers (exit_code=99) [production]
15:28 <legoktm@deploy1001> Finished scap: Rebuild l10n cache for SecureLinkFixer message (duration: 18m 51s) [production]
15:21 <elukey@cumin1001> START - Cookbook sre.kafka.roll-restart-brokers [production]
15:18 <jijiki> Disable puppet on mw1347 and mw2136, depool and pool back - T219150 [production]
15:13 <elukey> remove snakebite from buster-wikimedia (not needed anymore) [production]
15:09 <legoktm@deploy1001> Started scap: Rebuild l10n cache for SecureLinkFixer message [production]
15:06 <legoktm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable SecureLinkFixer everywhere (T200751) (duration: 00m 47s) [production]
14:47 <cdanis@deploy1001> Synchronized wmf-config/etcd.php: I17c55428 dbctl canary on mwdebug*, mw1261, mw1276 (duration: 00m 47s) [production]
14:36 <cdanis@deploy1001> Synchronized wmf-config/CommonSettings.php: Ie98a8d9e dbctl canary on mwdebug1001 (duration: 00m 47s) [production]
14:34 <cdanis@deploy1001> Synchronized wmf-config/etcd.php: Ie98a8d9e dbctl canary on mwdebug1001 (duration: 00m 47s) [production]
14:33 <cdanis@deploy1001> Synchronized docroot/noc/db.php: Ie98a8d9e dbctl canary on mwdebug1001 (duration: 00m 48s) [production]
14:14 <fsero> refreshing calico policy from code in eqiad [production]
14:13 <fsero> refreshing calico policy from code in codfw [production]
13:38 <marostegui> Move db2094:3315 from db2066 to db2128 - T228258 [production]
13:14 <@> helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-analytics' for release 'analytics' . [production]
13:13 <@> helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-analytics' for release 'analytics' . [production]
12:36 <marostegui@cumin1001> dbctl commit of MediaWiki config (dc=all), diff saved to 'https://phabricator.wikimedia.org/P8824', previous config saved to /var/cache/conftool/dbconfig/20190730-123630-marostegui.json [production]
12:21 <@> helmfile [EQIAD] Ran 'apply' command on namespace 'termbox' for release 'production' . [production]
12:15 <@> helmfile [CODFW] Ran 'apply' command on namespace 'termbox' for release 'production' . [production]
12:13 <@> helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'test' . [production]
12:13 <jbond42> while testing some changes on the puppet master a bad config caused a small blip in catalouge compilation [production]
12:09 <@> helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'staging' . [production]
11:34 <@> helmfile [EQIAD] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
11:31 <@> helmfile [CODFW] Ran 'apply' command on namespace 'cxserver' for release 'production' . [production]
11:30 <jijiki> Depool mw1348 and pool back [production]
11:28 <@> helmfile [STAGING] Ran 'apply' command on namespace 'cxserver' for release 'staging' . [production]
09:49 <elukey> upload python-snakebite to buster-wikimedia (rebuilt for buster from source) [production]
09:31 <akosiaris@> helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
09:27 <elukey> add thirdparty/cloudera to buster-wikimedia and import packages to it (pull from the jessie component) [production]
08:17 <marostegui> Stop MySQL on db2038 T227565 [production]
08:10 <marostegui> Remove db2038 from tendril and zarcillo T227565 [production]
08:04 <akosiaris> delete orespoolcounter{1,2}00{1,2} T227640 [production]
08:04 <akosiaris> revoke and deactivate orespoolcounter{1,2}00{1,2} T227640 [production]
07:30 <godog> bounce hhvm on mw1221 [production]
05:36 <marostegui> Disable puppet on cumin2001 to investigate a backups issue [production]
05:25 <tstarling@deploy1001> Synchronized php-1.34.0-wmf.15/includes/jobqueue/jobs/AssembleUploadChunksJob.php: T228929 (duration: 00m 46s) [production]
05:24 <tstarling@deploy1001> Synchronized php-1.34.0-wmf.15/includes/api/ApiUpload.php: T228929 (duration: 00m 47s) [production]
05:23 <tstarling@deploy1001> Synchronized php-1.34.0-wmf.15/includes/upload/UploadBase.php: T228929 (duration: 00m 48s) [production]
05:02 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove s8 ready only T227062 (duration: 00m 24s) [production]
05:01 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Switchover s8 master eqiad from db1071 to db1104 T227062 (duration: 00m 24s) [production]
05:00 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Set s8 on read-only T227062 (duration: 00m 26s) [production]
05:00 <marostegui> Starting s8 failover from db1071 to db1104 - T227062 [production]
04:48 <eileen> civicrm revision changed from 1d57aca19c to 218328b29d, config revision is 3f960c48f6 [production]
04:15 <marostegui> Start pre-steps for s8 primary master failover - T227062 [production]
02:37 <eileen> civicrm revision changed from 121feb5d53 to 1d57aca19c, config revision is 3f960c48f6 [production]
2019-07-29 §
23:37 <XioNoX> replace export policy BGP_Wikimedia_own_space with BGP_Wikimedia_no_dfz in ams [production]
23:35 <catrope@deploy1001> Synchronized php-1.34.0-wmf.15/extensions/GrowthExperiments/: Make welcome and discovery tours fully mutually exclusive (T229044) (duration: 00m 48s) [production]
23:26 <XioNoX> replace export policy BGP_Wikimedia_own_space with BGP_Wikimedia_no_dfz in ulsfo [production]