|
2017-03-28
ยง
|
| 20:21 |
<mutante> |
copper - puppet errors due to Failed resource /var/lib/docker/devicemapper ?? |
[production] |
| 20:19 |
<mutante> |
mwdebug1002 - same, was low on disk space, 'apt-get clean' freed > 3GB |
[production] |
| 20:18 |
<mutante> |
mwdebug1001 - was low on disk space, 'apt-get clean' - freed about 4GB |
[production] |
| 20:15 |
<mutante> |
mw1261 - depooled |
[production] |
| 20:14 |
<mutante> |
mw1261 runs with HHVM 3.18 - which seems to have a bug leading to a deadlock every 4-5 hours |
[production] |
| 20:14 |
<thcipriani@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.29.0-wmf.18 |
[production] |
| 20:13 |
<mutante> |
mw1261 HHVM crash as predicted by Moritz - ran sudo hhvm-dump-debug. Backtrace saved as /tmp/hhvm.79460.bt. |
[production] |
| 20:06 |
<mutante> |
ms-fe100[1-4] - disable/stop puppet, stop salt minion, decom (T160986) |
[production] |
| 19:57 |
<thcipriani@tin> |
Finished scap: testwiki to php-1.29.0-wmf.18 and rebuild l10n cache (duration: 40m 19s) |
[production] |
| 19:53 |
<hashar> |
Populating package manager cache of oojs-ui-npm-run-jenkins-node-6-jessie by manually triggering a build with ZUUL_PIPELINE=postmerge T155483 |
[releng] |
| 19:37 |
<mobrovac> |
restbase deploying d477f495 |
[production] |
| 19:34 |
<hashar> |
Migrate oojs/ui to just run 'npm jenkins' https://gerrit.wikimedia.org/r/345203 / T155483 |
[releng] |
| 19:33 |
<urandom> |
T111113: Restarting Cassandra instances, codfw row 'd' {{done}} |
[production] |
| 19:17 |
<thcipriani@tin> |
Started scap: testwiki to php-1.29.0-wmf.18 and rebuild l10n cache |
[production] |
| 18:45 |
<urandom> |
T111113: Restarting Cassandra instances, codfw row 'd' |
[production] |
| 18:44 |
<urandom> |
T111113: Restarting Cassandra instances, codfw row 'c' {{done}} |
[production] |
| 18:18 |
<ppchelko@tin> |
Finished deploy [changeprop/deploy@1689d86]: Rename event field in logs (duration: 00m 52s) |
[production] |
| 18:18 |
<ppchelko@tin> |
Started deploy [changeprop/deploy@1689d86]: Rename event field in logs |
[production] |
| 17:53 |
<urandom> |
T111113: Restarting Cassandra instances, codfw row 'c' |
[production] |
| 17:22 |
<thcipriani> |
starting branch cut for 1.29.0-wmf.18 |
[production] |
| 17:07 |
<godog> |
swift codfw-prod: bump ms-be2028 ms-be2039 object weight to 3000 - T158337 |
[production] |
| 17:06 |
<gehel@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=elastic2021.codfw.wmnet |
[production] |
| 16:39 |
<urandom> |
T111113: Restarting remaining Cassandra instances, rack 'b', codfw (restbase20{02,07,10}) |
[production] |
| 16:19 |
<urandom> |
T111113: Restarting Cassandra on restbase2001 to apply mandatory client encryption (canary) |
[production] |
| 16:05 |
<halfak> |
deployed ores:18beebf (T160638) |
[releng] |
| 15:58 |
<halfak> |
deployed ores-wmflabs-deploy:18beebf (T160638) |
[ores] |
| 15:57 |
<halfak> |
deployed ores-wmflabs-deploy:18beebf |
[ores] |
| 15:56 |
<gehel> |
banning elastic2021 to run same tests as elastic2020 - T149006 |
[production] |
| 15:48 |
<halfak> |
staged ores-wmflabs-deploy:18beebf |
[ores] |
| 14:41 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=mw2256.codfw.wmnet |
[production] |
| 14:40 |
<marostegui> |
Convert dewiki UNIQUE keys into PK on db1091 (commonswiki) - T17441 |
[production] |
| 14:38 |
<ppchelko@tin> |
Finished deploy [changeprop/deploy@bfbaa17]: Increase log level for processinng failures (duration: 01m 07s) |
[production] |
| 14:38 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1091 - T17441 (duration: 00m 43s) |
[production] |
| 14:38 |
<elukey> |
ran restart-hhvm on mw1242, hhvm threads stuck (dump debug in /tmp/hhvm.9008.bt.) - HHVM 3.12 |
[production] |
| 14:37 |
<ppchelko@tin> |
Started deploy [changeprop/deploy@bfbaa17]: Increase log level for processinng failures |
[production] |
| 14:30 |
<elukey> |
analytics1028 back serving traffic - T159632 |
[analytics] |
| 13:54 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1092 - T17441 (duration: 00m 43s) |
[production] |
| 13:44 |
<elukey> |
started hhvm on mw1261 (still depooled) - no hhvm process running |
[production] |
| 13:29 |
<RoanKattouw> |
Ran initUserPreference.php -s ores-enabled -t rcenhancedfilters and -s ores-enabled -t oresHighlight on plwiki and ptwiki |
[production] |
| 13:22 |
<catrope@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable RCFilters beta feature on plwiki and ptwiki T158336 (duration: 00m 43s) |
[production] |
| 13:22 |
<gehel> |
restarting elasticsearch on deployment-elastic05 to reload log4j configuration |
[releng] |
| 12:58 |
<moritzm> |
depooled mw1261 |
[production] |
| 10:39 |
<ema> |
upgrading twisted to 16.2.0 on lvs3001 and lvs3002 (esams primaries) T160433 |
[production] |
| 10:36 |
<ema> |
upgrading twisted to 16.2.0 on lvs3003 and lvs3004 (esams secondaries) T160433 |
[production] |
| 10:28 |
<hashar> |
Jenkins: installing Android Lint plugin 2.4 - T161305 |
[releng] |
| 10:27 |
<marostegui> |
Convert dewiki UNIQUE keys into PK on db1092 - https://phabricator.wikimedia.org/T17441 |
[production] |
| 10:14 |
<elukey> |
Switching hue.w.o's backend (cache misc) from anaytics1027 to thorium - T159527 |
[production] |
| 10:10 |
<moritzm> |
upgraded mw1262 to HHVM 3.18 |
[production] |
| 08:48 |
<marostegui> |
Convert wikidatawiki UNIQUE keys into PK on db1092 - T17441 |
[production] |
| 08:48 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1092 - T17441 (duration: 00m 44s) |
[production] |