|
2018-10-23
§
|
| 18:13 |
<mutante> |
icinga1001 - manually set max_concurrent_checks to 0 (unlimited), restart icinga, keep puppet disabled, for testing (it ran into the limit of 10000 all the time, causing lots of logging, and the CPU power is actually slightly lower than on einsteinium (T202782) refs: Nagios Tuning, point 7 https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/3/en/tuning.html |
[production] |
| 17:20 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: BETA: Set wmgWikibaseCachePrefix for commonswiki I0badd355723 (duration: 00m 46s) |
[production] |
| 17:18 |
<ejegg> |
updated standalone SmashPig deploy from 2292111bda to b638ca02bc |
[production] |
| 17:15 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: For WBMI, intentionally rather than implicitly install Wikibase I38574e670 (duration: 00m 47s) |
[production] |
| 17:13 |
<mutante> |
icinga1001 rm /var/log/user.log.1 - was 14G and using 25% of the / partition and server out of disk :/ |
[production] |
| 17:06 |
<ejegg> |
rolled SmashPig back to 2292111bda |
[production] |
| 17:03 |
<ejegg> |
updated standalone SmashPig deployment from 2292111bda to 18da9727d8 |
[production] |
| 16:20 |
<volans> |
restarted pdfrender on scb1004 |
[production] |
| 14:47 |
<herron> |
added confluent-kafka-2.11 1.1.0-1 package to jessie-wikimedia/thirdparty T206454 |
[production] |
| 14:34 |
<anomie@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Setting comment table migration stage to write-new/read-both on group 1 (T166733) (duration: 00m 46s) |
[production] |
| 14:22 |
<anomie@deploy1001> |
Synchronized php-1.32.0-wmf.26/includes/filerepo/file/LocalFile.php: Backport for T207419 (duration: 00m 47s) |
[production] |
| 14:02 |
<gehel> |
repooling / banning elastics1031 - T207724 |
[production] |
| 14:01 |
<moritzm> |
installing spice security updates |
[production] |
| 14:00 |
<ema> |
upload trafficserver 8.0.0-1wm1 to stretch-wikimedia/main T204232 |
[production] |
| 13:48 |
<gehel> |
depooling / banning elastics1031 - T207724 |
[production] |
| 13:43 |
<gehel> |
depooling / banning elastics1029 - T207724 |
[production] |
| 13:35 |
<gehel> |
rolling restart of blazegraph for change to blazegraph home dir |
[production] |
| 13:22 |
<gehel> |
depooling / banning elastics1018 - T207724 |
[production] |
| 12:29 |
<gehel> |
depooling / banning elastics1028 and 1030 - T207724 |
[production] |
| 11:23 |
<zeljkof> |
EU SWAT finished |
[production] |
| 11:20 |
<zfilipin@deploy1001> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:469168|New throttle rule for Wikipedia in Ort (T207714)]] (duration: 00m 46s) |
[production] |
| 11:11 |
<zfilipin@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:469180|Enable RCPatrol for srwikiquote (T207732)]] (duration: 00m 47s) |
[production] |
| 10:13 |
<ema> |
upload libc++ 6.0.1 to stretch-wikimedia/main T204232 |
[production] |
| 09:42 |
<jynus> |
stopping db1087 to fix db1124 |
[production] |
| 09:31 |
<gehel> |
depooling / banning elastics1017 and 1022 - T207724 |
[production] |
| 09:13 |
<godog> |
roll-restart thumbor to send statsd traffic through statsd_exporter - T205870 |
[production] |
| 08:08 |
<godog> |
update hp firmware to 6.60 on ms-be2017 - T141756 |
[production] |
| 07:14 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Repool db1100 - T184805 (duration: 00m 48s) |
[production] |
| 06:50 |
<elukey> |
powercycle ms-be2017 (frozen since ~8hrs ago) |
[production] |
| 06:42 |
<elukey> |
restart yarn and hdfs daemon on analytics1068 to pick up correct config (the host was down since before we swapped the Hadoop masters due to hw failure) |
[production] |
| 06:39 |
<marostegui> |
Stop replication on db1092 and db1087 for checking T206743 |
[production] |
| 06:02 |
<marostegui> |
Deploy schema change on s3 - T207359 |
[production] |
| 00:35 |
<SMalyshev> |
temp depooled wdq1003 to let it catch up |
[production] |
| 00:17 |
<Amir1> |
evening SWAT is done |
[production] |
|
2018-10-22
§
|
| 23:59 |
<ladsgroup@deploy1001> |
Synchronized php-1.32.0-wmf.26/includes/changetags/ChangeTags.php: SWAT: [[gerrit:469114|Fix bad join on ChangeTag subquery (T207313)]] (duration: 00m 47s) |
[production] |
| 23:39 |
<smalyshev@deploy1001> |
Finished deploy [wdqs/wdqs@d4692ea]: Redeploy Updater for T207673 (duration: 10m 12s) |
[production] |
| 23:29 |
<smalyshev@deploy1001> |
Started deploy [wdqs/wdqs@d4692ea]: Redeploy Updater for T207673 |
[production] |
| 22:12 |
<pmiazga@deploy1001> |
Synchronized wmf-config//InitialiseSettings-labs.php: SWAT: [[gerrit:469121|beta: Disable page issues A/B test on beta cluster only (T200792)]] (duration: 00m 46s) |
[production] |
| 21:44 |
<mutante> |
adding new prod ServerAlias punjabi.wikimedia.org to Apache cluster (T207583) |
[production] |
| 21:13 |
<ayounsi@deploy1001> |
Finished deploy [librenms/librenms@0fd8da6]: Revert LibreNMS upgrade - T207481 (duration: 00m 08s) |
[production] |
| 21:13 |
<ayounsi@deploy1001> |
Started deploy [librenms/librenms@0fd8da6]: Revert LibreNMS upgrade - T207481 |
[production] |
| 21:08 |
<andrewbogott> |
rebooting cloudvirt1023 |
[production] |
| 20:52 |
<ayounsi@deploy1001> |
Finished deploy [librenms/librenms@737683a]: Upgreade LibreNMS to 1.44 - T207481 (duration: 00m 10s) |
[production] |
| 20:52 |
<ayounsi@deploy1001> |
Started deploy [librenms/librenms@737683a]: Upgreade LibreNMS to 1.44 - T207481 |
[production] |
| 20:28 |
<ladsgroup@deploy1001> |
Finished deploy [ores/deploy@e89e880]: Use redis task tracker (T152012) (duration: 22m 02s) |
[production] |
| 20:06 |
<ladsgroup@deploy1001> |
Started deploy [ores/deploy@e89e880]: Use redis task tracker (T152012) |
[production] |
| 18:54 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT Deploy TemplateWizard everywhere T202545, re-try (duration: 00m 45s) |
[production] |
| 18:50 |
<jforrester@deploy1001> |
scap failed: average error rate on 4/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) |
[production] |
| 18:48 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: [Beta] Temporarily disable WBMI from Beta Commons whilst Wikibse is fixed T180981 (duration: 00m 46s) |
[production] |
| 18:38 |
<jforrester@deploy1001> |
Synchronized php-1.32.0-wmf.26/resources/src/mediawiki.rcfilters/styles/mw.rcfilters.ui.ChangesListWrapperWidget.highlightCircles.seenunseen.less: SWAT RCFIlters: Fix highlight circles for unseen changes T207472 (duration: 00m 46s) |
[production] |