|
2017-03-28
ยง
|
| 21:08 |
<urandom> |
T111113: Restarting Cassandra instances, eqiad row 'a' {{done}} |
[production] |
| 20:24 |
<mutante> |
ms-fe1001 thru msfe1004 - scheduled last downtime for host and services in icinga - shutdown -h now, turn them off, revoke puppet certs, salt-keys... (T160986) |
[production] |
| 20:22 |
<mutante> |
mc1019 - puppet fail due to Failed resource /etc/redis/replica since 4 days |
[production] |
| 20:21 |
<urandom> |
T111113: Restarting Cassandra instances, eqiad row 'a' |
[production] |
| 20:21 |
<mutante> |
copper - puppet errors due to Failed resource /var/lib/docker/devicemapper ?? |
[production] |
| 20:19 |
<mutante> |
mwdebug1002 - same, was low on disk space, 'apt-get clean' freed > 3GB |
[production] |
| 20:18 |
<mutante> |
mwdebug1001 - was low on disk space, 'apt-get clean' - freed about 4GB |
[production] |
| 20:15 |
<mutante> |
mw1261 - depooled |
[production] |
| 20:14 |
<mutante> |
mw1261 runs with HHVM 3.18 - which seems to have a bug leading to a deadlock every 4-5 hours |
[production] |
| 20:14 |
<thcipriani@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.29.0-wmf.18 |
[production] |
| 20:13 |
<mutante> |
mw1261 HHVM crash as predicted by Moritz - ran sudo hhvm-dump-debug. Backtrace saved as /tmp/hhvm.79460.bt. |
[production] |
| 20:06 |
<mutante> |
ms-fe100[1-4] - disable/stop puppet, stop salt minion, decom (T160986) |
[production] |
| 19:57 |
<thcipriani@tin> |
Finished scap: testwiki to php-1.29.0-wmf.18 and rebuild l10n cache (duration: 40m 19s) |
[production] |
| 19:37 |
<mobrovac> |
restbase deploying d477f495 |
[production] |
| 19:33 |
<urandom> |
T111113: Restarting Cassandra instances, codfw row 'd' {{done}} |
[production] |
| 19:17 |
<thcipriani@tin> |
Started scap: testwiki to php-1.29.0-wmf.18 and rebuild l10n cache |
[production] |
| 18:45 |
<urandom> |
T111113: Restarting Cassandra instances, codfw row 'd' |
[production] |
| 18:44 |
<urandom> |
T111113: Restarting Cassandra instances, codfw row 'c' {{done}} |
[production] |
| 18:18 |
<ppchelko@tin> |
Finished deploy [changeprop/deploy@1689d86]: Rename event field in logs (duration: 00m 52s) |
[production] |
| 18:18 |
<ppchelko@tin> |
Started deploy [changeprop/deploy@1689d86]: Rename event field in logs |
[production] |
| 17:53 |
<urandom> |
T111113: Restarting Cassandra instances, codfw row 'c' |
[production] |
| 17:22 |
<thcipriani> |
starting branch cut for 1.29.0-wmf.18 |
[production] |
| 17:07 |
<godog> |
swift codfw-prod: bump ms-be2028 ms-be2039 object weight to 3000 - T158337 |
[production] |
| 17:06 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=elastic2021.codfw.wmnet |
[production] |
| 16:39 |
<urandom> |
T111113: Restarting remaining Cassandra instances, rack 'b', codfw (restbase20{02,07,10}) |
[production] |
| 16:19 |
<urandom> |
T111113: Restarting Cassandra on restbase2001 to apply mandatory client encryption (canary) |
[production] |
| 15:56 |
<gehel> |
banning elastic2021 to run same tests as elastic2020 - T149006 |
[production] |
| 14:41 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=mw2256.codfw.wmnet |
[production] |
| 14:40 |
<marostegui> |
Convert dewiki UNIQUE keys into PK on db1091 (commonswiki) - T17441 |
[production] |
| 14:38 |
<ppchelko@tin> |
Finished deploy [changeprop/deploy@bfbaa17]: Increase log level for processinng failures (duration: 01m 07s) |
[production] |
| 14:38 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1091 - T17441 (duration: 00m 43s) |
[production] |
| 14:38 |
<elukey> |
ran restart-hhvm on mw1242, hhvm threads stuck (dump debug in /tmp/hhvm.9008.bt.) - HHVM 3.12 |
[production] |
| 14:37 |
<ppchelko@tin> |
Started deploy [changeprop/deploy@bfbaa17]: Increase log level for processinng failures |
[production] |
| 13:54 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1092 - T17441 (duration: 00m 43s) |
[production] |
| 13:44 |
<elukey> |
started hhvm on mw1261 (still depooled) - no hhvm process running |
[production] |
| 13:29 |
<RoanKattouw> |
Ran initUserPreference.php -s ores-enabled -t rcenhancedfilters and -s ores-enabled -t oresHighlight on plwiki and ptwiki |
[production] |
| 13:22 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable RCFilters beta feature on plwiki and ptwiki T158336 (duration: 00m 43s) |
[production] |
| 12:58 |
<moritzm> |
depooled mw1261 |
[production] |
| 10:39 |
<ema> |
upgrading twisted to 16.2.0 on lvs3001 and lvs3002 (esams primaries) T160433 |
[production] |
| 10:36 |
<ema> |
upgrading twisted to 16.2.0 on lvs3003 and lvs3004 (esams secondaries) T160433 |
[production] |
| 10:27 |
<marostegui> |
Convert dewiki UNIQUE keys into PK on db1092 - https://phabricator.wikimedia.org/T17441 |
[production] |
| 10:14 |
<elukey> |
Switching hue.w.o's backend (cache misc) from anaytics1027 to thorium - T159527 |
[production] |
| 10:10 |
<moritzm> |
upgraded mw1262 to HHVM 3.18 |
[production] |
| 08:48 |
<marostegui> |
Convert wikidatawiki UNIQUE keys into PK on db1092 - T17441 |
[production] |
| 08:48 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1092 - T17441 (duration: 00m 44s) |
[production] |
| 08:29 |
<akosiaris> |
enable IGMP snooping on all VLANs on asw2-d-eqiad. T133387 |
[production] |
| 07:19 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1089 - T17441 (duration: 00m 43s) |
[production] |
| 07:18 |
<moritzm> |
installing eject security updates on trusty hosts |
[production] |
| 06:11 |
<marostegui> |
Keep converting unique keys into PK on db1089 - T17441 |
[production] |
| 06:01 |
<marostegui> |
Deploy schema change on s2.enwiktionary.templatelinks - on codfw master, this will generate lag on codfw slaves (which have been silenced) - T154097 |
[production] |