|
2018-10-23
§
|
| 17:13 |
<mutante> |
icinga1001 rm /var/log/user.log.1 - was 14G and using 25% of the / partition and server out of disk :/ |
[production] |
| 17:06 |
<ejegg> |
rolled SmashPig back to 2292111bda |
[production] |
| 17:03 |
<ejegg> |
updated standalone SmashPig deployment from 2292111bda to 18da9727d8 |
[production] |
| 16:20 |
<volans> |
restarted pdfrender on scb1004 |
[production] |
| 14:47 |
<herron> |
added confluent-kafka-2.11 1.1.0-1 package to jessie-wikimedia/thirdparty T206454 |
[production] |
| 14:34 |
<anomie@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Setting comment table migration stage to write-new/read-both on group 1 (T166733) (duration: 00m 46s) |
[production] |
| 14:22 |
<anomie@deploy1001> |
Synchronized php-1.32.0-wmf.26/includes/filerepo/file/LocalFile.php: Backport for T207419 (duration: 00m 47s) |
[production] |
| 14:02 |
<gehel> |
repooling / banning elastics1031 - T207724 |
[production] |
| 14:01 |
<moritzm> |
installing spice security updates |
[production] |
| 14:00 |
<ema> |
upload trafficserver 8.0.0-1wm1 to stretch-wikimedia/main T204232 |
[production] |
| 13:48 |
<gehel> |
depooling / banning elastics1031 - T207724 |
[production] |
| 13:43 |
<gehel> |
depooling / banning elastics1029 - T207724 |
[production] |
| 13:35 |
<gehel> |
rolling restart of blazegraph for change to blazegraph home dir |
[production] |
| 13:22 |
<gehel> |
depooling / banning elastics1018 - T207724 |
[production] |
| 12:29 |
<gehel> |
depooling / banning elastics1028 and 1030 - T207724 |
[production] |
| 11:23 |
<zeljkof> |
EU SWAT finished |
[production] |
| 11:20 |
<zfilipin@deploy1001> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:469168|New throttle rule for Wikipedia in Ort (T207714)]] (duration: 00m 46s) |
[production] |
| 11:11 |
<zfilipin@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:469180|Enable RCPatrol for srwikiquote (T207732)]] (duration: 00m 47s) |
[production] |
| 10:13 |
<ema> |
upload libc++ 6.0.1 to stretch-wikimedia/main T204232 |
[production] |
| 09:42 |
<jynus> |
stopping db1087 to fix db1124 |
[production] |
| 09:31 |
<gehel> |
depooling / banning elastics1017 and 1022 - T207724 |
[production] |
| 09:13 |
<godog> |
roll-restart thumbor to send statsd traffic through statsd_exporter - T205870 |
[production] |
| 08:08 |
<godog> |
update hp firmware to 6.60 on ms-be2017 - T141756 |
[production] |
| 07:14 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1100 - T184805 (duration: 00m 48s) |
[production] |
| 06:50 |
<elukey> |
powercycle ms-be2017 (frozen since ~8hrs ago) |
[production] |
| 06:42 |
<elukey> |
restart yarn and hdfs daemon on analytics1068 to pick up correct config (the host was down since before we swapped the Hadoop masters due to hw failure) |
[production] |
| 06:39 |
<marostegui> |
Stop replication on db1092 and db1087 for checking T206743 |
[production] |
| 06:02 |
<marostegui> |
Deploy schema change on s3 - T207359 |
[production] |
| 00:35 |
<SMalyshev> |
temp depooled wdq1003 to let it catch up |
[production] |
| 00:17 |
<Amir1> |
evening SWAT is done |
[production] |
|
2018-10-22
§
|
| 23:59 |
<ladsgroup@deploy1001> |
Synchronized php-1.32.0-wmf.26/includes/changetags/ChangeTags.php: SWAT: [[gerrit:469114|Fix bad join on ChangeTag subquery (T207313)]] (duration: 00m 47s) |
[production] |
| 23:39 |
<smalyshev@deploy1001> |
Finished deploy [wdqs/wdqs@d4692ea]: Redeploy Updater for T207673 (duration: 10m 12s) |
[production] |
| 23:29 |
<smalyshev@deploy1001> |
Started deploy [wdqs/wdqs@d4692ea]: Redeploy Updater for T207673 |
[production] |
| 22:12 |
<pmiazga@deploy1001> |
Synchronized wmf-config//InitialiseSettings-labs.php: SWAT: [[gerrit:469121|beta: Disable page issues A/B test on beta cluster only (T200792)]] (duration: 00m 46s) |
[production] |
| 21:44 |
<mutante> |
adding new prod ServerAlias punjabi.wikimedia.org to Apache cluster (T207583) |
[production] |
| 21:13 |
<ayounsi@deploy1001> |
Finished deploy [librenms/librenms@0fd8da6]: Revert LibreNMS upgrade - T207481 (duration: 00m 08s) |
[production] |
| 21:13 |
<ayounsi@deploy1001> |
Started deploy [librenms/librenms@0fd8da6]: Revert LibreNMS upgrade - T207481 |
[production] |
| 21:08 |
<andrewbogott> |
rebooting cloudvirt1023 |
[production] |
| 20:52 |
<ayounsi@deploy1001> |
Finished deploy [librenms/librenms@737683a]: Upgreade LibreNMS to 1.44 - T207481 (duration: 00m 10s) |
[production] |
| 20:52 |
<ayounsi@deploy1001> |
Started deploy [librenms/librenms@737683a]: Upgreade LibreNMS to 1.44 - T207481 |
[production] |
| 20:28 |
<ladsgroup@deploy1001> |
Finished deploy [ores/deploy@e89e880]: Use redis task tracker (T152012) (duration: 22m 02s) |
[production] |
| 20:06 |
<ladsgroup@deploy1001> |
Started deploy [ores/deploy@e89e880]: Use redis task tracker (T152012) |
[production] |
| 18:54 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT Deploy TemplateWizard everywhere T202545, re-try (duration: 00m 45s) |
[production] |
| 18:50 |
<jforrester@deploy1001> |
scap failed: average error rate on 4/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) |
[production] |
| 18:48 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: [Beta] Temporarily disable WBMI from Beta Commons whilst Wikibse is fixed T180981 (duration: 00m 46s) |
[production] |
| 18:38 |
<jforrester@deploy1001> |
Synchronized php-1.32.0-wmf.26/resources/src/mediawiki.rcfilters/styles/mw.rcfilters.ui.ChangesListWrapperWidget.highlightCircles.seenunseen.less: SWAT RCFIlters: Fix highlight circles for unseen changes T207472 (duration: 00m 46s) |
[production] |
| 18:36 |
<jforrester@deploy1001> |
Synchronized php-1.32.0-wmf.26/skins/MinervaNeue/resources/skins.minerva.scripts/pageIssuesLogger.js: SWAT Fix reading depth logging part 2 T207423 (duration: 00m 46s) |
[production] |
| 18:35 |
<jforrester@deploy1001> |
Synchronized php-1.32.0-wmf.26/extensions/WikimediaEvents/modules/all/ext.wikimediaEvents.readingDepth.js: SWAT Fix reading depth logging part 1 T207423 (duration: 00m 46s) |
[production] |
| 18:31 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT Add TemplateWizard to the BF allow list T205290 (duration: 00m 48s) |
[production] |
| 18:05 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: [Beta Cluster] Load but don't enable MediaInfo on Beta Commons cf. T180981 (duration: 00m 45s) |
[production] |