4301-4350 of 10000 results (64ms)
2018-10-25 §
11:08 <zfilipin@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:465418|Stop collecting data CitaitonUsage and CitationUsagePageLoad (T191086 T203253)]] (duration: 00m 57s) [production]
10:57 <volans> restart pdfrender on scb1003 [production]
10:11 <elukey> upgrade druid100[1-3] to druid 0.12.3 [production]
09:51 <gehel> resetting deployment directory on wdqs1003 [production]
09:15 <elukey@deploy1001> Finished deploy [analytics/turnilo/deploy@84bf1ad]: Upgrade to 1.8.1 (duration: 00m 10s) [production]
09:15 <elukey@deploy1001> Started deploy [analytics/turnilo/deploy@84bf1ad]: Upgrade to 1.8.1 [production]
09:10 <ema> resume cache hosts rolling reboots for kernel/microcode updates T203011 [production]
07:16 <vgutierrez> Uploaded certcentral 0.3 to apt.wikimedia.org (stretch) - T207737 T207478 [production]
07:11 <moritzm> installing requests security updates on trusty [production]
06:17 <SMalyshev> depooling wdqs1003 again, it's not catching up like the other hosts [production]
06:06 <elukey> upload druid 0.12.3-1 debs to stretch-wikimedia [production]
2018-10-24 §
23:24 <maxsem@deploy1001> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/469495/ (duration: 00m 54s) [production]
23:15 <maxsem@deploy1001> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/462040/ (duration: 00m 55s) [production]
23:08 <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Deploy csp report-only to small.dblist wikis T207900 (duration: 00m 56s) [production]
22:38 <bawolff@deploy1001> Synchronized wmf-config/CommonSettings.php: Deploy csp report-only to outreachwiki T207900 (duration: 00m 54s) [production]
22:36 <bawolff@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Deploy csp report-only to outreachwiki T207900 (duration: 00m 54s) [production]
22:33 <bawolff@deploy1001> scap failed: average error rate on 8/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) [production]
22:27 <eileen_> civicrm revision changed from 1c0a1b2406 to 97506677e8, config revision is c0a8be03a1 [production]
21:33 <banyek> compressing tables in s1@dbstore2002 (T204930) [production]
21:26 <banyek> pausing replication on dbstore2002 (T204930) [production]
19:38 <twentyafterfour> The train is now blocked by database lock contention of unknown origin [production]
19:31 <twentyafterfour> the errors were all coming from wmf.26 but the error rate skyrocketed after deploying 1.33.0-wmf.1 to group1 so there is some query in the new branch which is holding a lock. T207881 [production]
19:19 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.1 refs T206655 [production]
18:16 <XioNoX> enable BGP sessions to transit/peering on cr2-eqord - T204170 [production]
17:20 <gehel> repooling all elasticsearch servers in eqiad [production]
17:12 <cmjohnson1> rebooting cloudvirt1019 [production]
17:04 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: [Beta Cluster] Re-disable WBMI on Beta Commons for now T180981 (duration: 00m 54s) [production]
17:03 <jforrester@deploy1001> scap failed: average error rate on 4/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) [production]
16:36 <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: [Beta Cluster] Re-disable WBMI on Beta Commons for now T180981 (duration: 00m 54s) [production]
16:31 <addshore@deploy1001> Synchronized wmf-config/Wikibase.php: [[gerrit:469444]] Wikibase.php, dont load wikidata repo settings on other repos (take 2) (duration: 00m 54s) [production]
16:04 <XioNoX> power-off cr1-eqord - T204170 [production]
16:00 <twentyafterfour> 15:59:06 Synchronized php-1.33.0-wmf.1/extensions/EventBus/: revert "Set event datetime with microsecond resolution." on 1.33.0-wmf.1 refs T207817 (duration: 00m 56s) [production]
15:59 <XioNoX> disable BGP sessions to transit/peering on cr1-eqord - T204170 [production]
15:54 <twentyafterfour> deploying https://gerrit.wikimedia.org/r/469451 [production]
14:23 <herron> scheduled icinga downtime and disabling puppet on logstash hosts. deploying role::kafka::logging to logstash elasticserach data hosts [production]
13:35 <XioNoX> pre-configure switch ports for labvirt1007/8/9/12:eth1 in cloud-virt-instance-trunk range on asw2-b-eqiad [production]
13:17 <ema> begin cache hosts rolling reboots for kernel/microcode updates T203011 [production]
12:24 <ema> cp-ats: upgrade trafficserver to 8.0.0-1wm1 T204232 [production]
12:12 <ema> cp1072: upgrade trafficserver to 8.0.0-1wm1 T204232 [production]
11:22 <ema> cp1071: upgrade trafficserver to 8.0.0-1wm1 T204232 [production]
10:56 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Restore db1092 and db1104 original weight (duration: 00m 52s) [production]
10:35 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1092 and starting to restore db1104 original weight (duration: 00m 54s) [production]
10:28 <marostegui> Compare revision table on dewiki cebwiki shwiki srwiki mgwiktionary enwikivoyage on db1100 and db2075 - T184805 [production]
09:54 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1092 (duration: 00m 54s) [production]
09:39 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1092 (duration: 00m 54s) [production]
09:20 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool db1092 and db1087 (duration: 01m 05s) [production]
08:55 <marostegui> Stop MySQL for upgrade and reboot on db1087 [production]
08:47 <marostegui> Update MySQL on db1092 for upgrade and reboot [production]
08:03 <godog> fix aggregation to 'sum' for MediaWiki.RevisionSlider - T205416 [production]
07:33 <gehel> powercycling wdqs1010 - T207817 [production]