|
2018-10-25
§
|
| 14:20 |
<banyek> |
running dns update (gerrit patch: 467711) |
[production] |
| 13:48 |
<anomie@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Setting comment table migration stage to write-new/read-both on all wikis (T166733) (duration: 00m 55s) |
[production] |
| 13:46 |
<godog> |
reformat ms-be2043 xfs filesystems - T199198 |
[production] |
| 13:29 |
<XioNoX> |
test successful, rollback add term return-tcp permit on cr2-codfw |
[production] |
| 13:28 |
<XioNoX> |
test add term return-tcp permit on cr2-codfw |
[production] |
| 12:14 |
<volans> |
rebooting cumin1001 to pick new kernel and clear any potential weird state after OOMs |
[production] |
| 12:01 |
<zeljkof> |
EU SWAT finished |
[production] |
| 11:17 |
<zfilipin@deploy1001> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:469261|New throttle rule for Johannesburg Event on 2018-10-27 (T207742)]] (duration: 00m 55s) |
[production] |
| 11:08 |
<zfilipin@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:465418|Stop collecting data CitaitonUsage and CitationUsagePageLoad (T191086 T203253)]] (duration: 00m 57s) |
[production] |
| 10:57 |
<volans> |
restart pdfrender on scb1003 |
[production] |
| 10:11 |
<elukey> |
upgrade druid100[1-3] to druid 0.12.3 |
[production] |
| 09:51 |
<gehel> |
resetting deployment directory on wdqs1003 |
[production] |
| 09:15 |
<elukey@deploy1001> |
Finished deploy [analytics/turnilo/deploy@84bf1ad]: Upgrade to 1.8.1 (duration: 00m 10s) |
[production] |
| 09:15 |
<elukey@deploy1001> |
Started deploy [analytics/turnilo/deploy@84bf1ad]: Upgrade to 1.8.1 |
[production] |
| 09:10 |
<ema> |
resume cache hosts rolling reboots for kernel/microcode updates T203011 |
[production] |
| 07:16 |
<vgutierrez> |
Uploaded certcentral 0.3 to apt.wikimedia.org (stretch) - T207737 T207478 |
[production] |
| 07:11 |
<moritzm> |
installing requests security updates on trusty |
[production] |
| 06:17 |
<SMalyshev> |
depooling wdqs1003 again, it's not catching up like the other hosts |
[production] |
| 06:06 |
<elukey> |
upload druid 0.12.3-1 debs to stretch-wikimedia |
[production] |
|
2018-10-24
§
|
| 23:24 |
<maxsem@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/469495/ (duration: 00m 54s) |
[production] |
| 23:15 |
<maxsem@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/462040/ (duration: 00m 55s) |
[production] |
| 23:08 |
<bawolff@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Deploy csp report-only to small.dblist wikis T207900 (duration: 00m 56s) |
[production] |
| 22:38 |
<bawolff@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Deploy csp report-only to outreachwiki T207900 (duration: 00m 54s) |
[production] |
| 22:36 |
<bawolff@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Deploy csp report-only to outreachwiki T207900 (duration: 00m 54s) |
[production] |
| 22:33 |
<bawolff@deploy1001> |
scap failed: average error rate on 8/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) |
[production] |
| 22:27 |
<eileen_> |
civicrm revision changed from 1c0a1b2406 to 97506677e8, config revision is c0a8be03a1 |
[production] |
| 21:33 |
<banyek> |
compressing tables in s1@dbstore2002 (T204930) |
[production] |
| 21:26 |
<banyek> |
pausing replication on dbstore2002 (T204930) |
[production] |
| 19:38 |
<twentyafterfour> |
The train is now blocked by database lock contention of unknown origin |
[production] |
| 19:31 |
<twentyafterfour> |
the errors were all coming from wmf.26 but the error rate skyrocketed after deploying 1.33.0-wmf.1 to group1 so there is some query in the new branch which is holding a lock. T207881 |
[production] |
| 19:19 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.1 refs T206655 |
[production] |
| 18:16 |
<XioNoX> |
enable BGP sessions to transit/peering on cr2-eqord - T204170 |
[production] |
| 17:20 |
<gehel> |
repooling all elasticsearch servers in eqiad |
[production] |
| 17:12 |
<cmjohnson1> |
rebooting cloudvirt1019 |
[production] |
| 17:04 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: [Beta Cluster] Re-disable WBMI on Beta Commons for now T180981 (duration: 00m 54s) |
[production] |
| 17:03 |
<jforrester@deploy1001> |
scap failed: average error rate on 4/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) |
[production] |
| 16:36 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: [Beta Cluster] Re-disable WBMI on Beta Commons for now T180981 (duration: 00m 54s) |
[production] |
| 16:31 |
<addshore@deploy1001> |
Synchronized wmf-config/Wikibase.php: [[gerrit:469444]] Wikibase.php, dont load wikidata repo settings on other repos (take 2) (duration: 00m 54s) |
[production] |
| 16:04 |
<XioNoX> |
power-off cr1-eqord - T204170 |
[production] |
| 16:00 |
<twentyafterfour> |
15:59:06 Synchronized php-1.33.0-wmf.1/extensions/EventBus/: revert "Set event datetime with microsecond resolution." on 1.33.0-wmf.1 refs T207817 (duration: 00m 56s) |
[production] |
| 15:59 |
<XioNoX> |
disable BGP sessions to transit/peering on cr1-eqord - T204170 |
[production] |
| 15:54 |
<twentyafterfour> |
deploying https://gerrit.wikimedia.org/r/469451 |
[production] |
| 14:23 |
<herron> |
scheduled icinga downtime and disabling puppet on logstash hosts. deploying role::kafka::logging to logstash elasticserach data hosts |
[production] |
| 13:35 |
<XioNoX> |
pre-configure switch ports for labvirt1007/8/9/12:eth1 in cloud-virt-instance-trunk range on asw2-b-eqiad |
[production] |
| 13:17 |
<ema> |
begin cache hosts rolling reboots for kernel/microcode updates T203011 |
[production] |
| 12:24 |
<ema> |
cp-ats: upgrade trafficserver to 8.0.0-1wm1 T204232 |
[production] |
| 12:12 |
<ema> |
cp1072: upgrade trafficserver to 8.0.0-1wm1 T204232 |
[production] |
| 11:22 |
<ema> |
cp1071: upgrade trafficserver to 8.0.0-1wm1 T204232 |
[production] |
| 10:56 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Restore db1092 and db1104 original weight (duration: 00m 52s) |
[production] |
| 10:35 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1092 and starting to restore db1104 original weight (duration: 00m 54s) |
[production] |