4101-4150 of 10000 results (63ms)
2018-10-23 §
09:31 <gehel> depooling / banning elastics1017 and 1022 - T207724 [production]
09:13 <godog> roll-restart thumbor to send statsd traffic through statsd_exporter - T205870 [production]
08:08 <godog> update hp firmware to 6.60 on ms-be2017 - T141756 [production]
07:14 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool db1100 - T184805 (duration: 00m 48s) [production]
06:50 <elukey> powercycle ms-be2017 (frozen since ~8hrs ago) [production]
06:42 <elukey> restart yarn and hdfs daemon on analytics1068 to pick up correct config (the host was down since before we swapped the Hadoop masters due to hw failure) [production]
06:39 <marostegui> Stop replication on db1092 and db1087 for checking T206743 [production]
06:02 <marostegui> Deploy schema change on s3 - T207359 [production]
00:35 <SMalyshev> temp depooled wdq1003 to let it catch up [production]
00:17 <Amir1> evening SWAT is done [production]
2018-10-22 §
23:59 <ladsgroup@deploy1001> Synchronized php-1.32.0-wmf.26/includes/changetags/ChangeTags.php: SWAT: [[gerrit:469114|Fix bad join on ChangeTag subquery (T207313)]] (duration: 00m 47s) [production]
23:39 <smalyshev@deploy1001> Finished deploy [wdqs/wdqs@d4692ea]: Redeploy Updater for T207673 (duration: 10m 12s) [production]
23:29 <smalyshev@deploy1001> Started deploy [wdqs/wdqs@d4692ea]: Redeploy Updater for T207673 [production]
22:12 <pmiazga@deploy1001> Synchronized wmf-config//InitialiseSettings-labs.php: SWAT: [[gerrit:469121|beta: Disable page issues A/B test on beta cluster only (T200792)]] (duration: 00m 46s) [production]
21:44 <mutante> adding new prod ServerAlias punjabi.wikimedia.org to Apache cluster (T207583) [production]
21:13 <ayounsi@deploy1001> Finished deploy [librenms/librenms@0fd8da6]: Revert LibreNMS upgrade - T207481 (duration: 00m 08s) [production]
21:13 <ayounsi@deploy1001> Started deploy [librenms/librenms@0fd8da6]: Revert LibreNMS upgrade - T207481 [production]
21:08 <andrewbogott> rebooting cloudvirt1023 [production]
20:52 <ayounsi@deploy1001> Finished deploy [librenms/librenms@737683a]: Upgreade LibreNMS to 1.44 - T207481 (duration: 00m 10s) [production]
20:52 <ayounsi@deploy1001> Started deploy [librenms/librenms@737683a]: Upgreade LibreNMS to 1.44 - T207481 [production]
20:28 <ladsgroup@deploy1001> Finished deploy [ores/deploy@e89e880]: Use redis task tracker (T152012) (duration: 22m 02s) [production]
20:06 <ladsgroup@deploy1001> Started deploy [ores/deploy@e89e880]: Use redis task tracker (T152012) [production]
18:54 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT Deploy TemplateWizard everywhere T202545, re-try (duration: 00m 45s) [production]
18:50 <jforrester@deploy1001> scap failed: average error rate on 4/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) [production]
18:48 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: [Beta] Temporarily disable WBMI from Beta Commons whilst Wikibse is fixed T180981 (duration: 00m 46s) [production]
18:38 <jforrester@deploy1001> Synchronized php-1.32.0-wmf.26/resources/src/mediawiki.rcfilters/styles/mw.rcfilters.ui.ChangesListWrapperWidget.highlightCircles.seenunseen.less: SWAT RCFIlters: Fix highlight circles for unseen changes T207472 (duration: 00m 46s) [production]
18:36 <jforrester@deploy1001> Synchronized php-1.32.0-wmf.26/skins/MinervaNeue/resources/skins.minerva.scripts/pageIssuesLogger.js: SWAT Fix reading depth logging part 2 T207423 (duration: 00m 46s) [production]
18:35 <jforrester@deploy1001> Synchronized php-1.32.0-wmf.26/extensions/WikimediaEvents/modules/all/ext.wikimediaEvents.readingDepth.js: SWAT Fix reading depth logging part 1 T207423 (duration: 00m 46s) [production]
18:31 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT Add TemplateWizard to the BF allow list T205290 (duration: 00m 48s) [production]
18:05 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: [Beta Cluster] Load but don't enable MediaInfo on Beta Commons cf. T180981 (duration: 00m 45s) [production]
18:00 <jforrester@deploy1001> Synchronized wmf-config/Wikibase.php: For WikibaseMediaInfo wikis, load basic Wikibase repo code cf. T180981 (duration: 00m 46s) [production]
17:56 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Allow enablement of the WikibaseMediaInfo, still off everywhere cf. T180981 (duration: 00m 48s) [production]
17:46 <jforrester@deploy1001> Synchronized wmf-config/extension-list: Add WikibaseMediaInfo i18n to cache cf. T180981 (duration: 00m 46s) [production]
17:40 <mobrovac@deploy1001> Finished deploy [proton/deploy@b3e254a]: Update Puppeteer to v1.9.0 - T207416 (duration: 01m 34s) [production]
17:38 <mobrovac@deploy1001> Started deploy [proton/deploy@b3e254a]: Update Puppeteer to v1.9.0 - T207416 [production]
17:34 <onimisionipe@deploy1001> Finished deploy [wdqs/wdqs@975a67b]: WDQS deployment - GUI update and binaries upgrade (duration: 11m 47s) [production]
17:23 <XioNoX> enable cr2:xe-4/0/0 (to asw-a) for optics replacement - T203719 [production]
17:22 <onimisionipe@deploy1001> Started deploy [wdqs/wdqs@975a67b]: WDQS deployment - GUI update and binaries upgrade [production]
17:20 <XioNoX> disable cr2:xe-4/0/0 (to asw-a) for optics replacement - T203719 [production]
17:19 <elukey@deploy1001> Finished deploy [analytics/refinery@1de5f44]: Deploy new version of Camus and pageview whitelist (duration: 07m 05s) [production]
17:16 <cmjohnson1> analytics1068 down for mother board swap [production]
17:12 <elukey@deploy1001> Started deploy [analytics/refinery@1de5f44]: Deploy new version of Camus and pageview whitelist [production]
17:11 <XioNoX> re-enable puppet fleet-wide for puppetmaster1001 uplink move [production]
17:10 <XioNoX> moving puppetmaster1001 uplink to asw2-b [production]
17:08 <andrew@deploy1001> Finished deploy [horizon/deploy@431a55d]: Rolling out fix for T207510 (duration: 03m 40s) [production]
17:07 <XioNoX> disable puppet fleet-wide for puppetmaster1001 uplink move [production]
17:05 <andrew@deploy1001> Started deploy [horizon/deploy@431a55d]: Rolling out fix for T207510 [production]
16:24 <arturo> T206261 2h icinga downtime cloudnet1003/4 for another patch [production]
15:54 <ejegg> updated payments-wiki from 06848600ed to 5a7a8e7e4f [production]
15:51 <ejegg> updated fundraising CiviCRM from 1f10dc8a18 to bde28d4453 [production]