|
2017-07-25
ยง
|
| 17:55 |
<demon@tin> |
Started scap: bootstrap wmf.11 (x2) |
[production] |
| 17:54 |
<demon@tin> |
scap failed: RuntimeError scap failed: average error rate on 1/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) (duration: 16m 32s) |
[production] |
| 17:53 |
<demon@tin> |
scap failed: average error rate on 1/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) |
[production] |
| 17:49 |
<urandom> |
Rolling restart of codfw Cassandra instances (applying OpenJDK update) |
[production] |
| 17:40 |
<halfak@tin> |
Finished deploy [ores/deploy@835d848]: T171505 (duration: 34m 56s) |
[production] |
| 17:37 |
<demon@tin> |
Started scap: bootstrap wmf.11 |
[production] |
| 17:33 |
<jynus> |
creating new database on m1 (rddmarc) T170158 |
[production] |
| 17:10 |
<demon@tin> |
Pruned MediaWiki: 1.30.0-wmf.9 [keeping static files] (duration: 01m 39s) |
[production] |
| 17:05 |
<halfak@tin> |
Started deploy [ores/deploy@835d848]: T171505 |
[production] |
| 16:16 |
<jynus> |
about to delete orfphan files on einstenium T149557 |
[production] |
| 15:49 |
<moritzm> |
installing imagemagick security updates on trusty hosts (jessie already fixed) |
[production] |
| 15:31 |
<cmjohnson1> |
updating firmware lvs1007 T167299 |
[production] |
| 15:03 |
<oblivian@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=restbase1015.eqiad.wmnet |
[production] |
| 14:58 |
<_joe_> |
enabled hyperthreading on restbase1015.eqiad.wmnet T162735, rebooting the server |
[production] |
| 14:52 |
<_joe_> |
shutting down restbase1015, T162735 |
[production] |
| 14:51 |
<oblivian@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=restbase1015.eqiad.wmnet |
[production] |
| 14:36 |
<urandom> |
draining restbase1015.eqiad.wmnet T162735 |
[production] |
| 14:35 |
<oblivian@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=restbase2002.codfw.wmnet |
[production] |
| 14:29 |
<_joe_> |
enabled hyperthreading on restbase2002.codfw.wmnet T162735, rebooting the server |
[production] |
| 14:23 |
<_joe_> |
shutting down restbase2002, T162735 |
[production] |
| 14:14 |
<oblivian@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=restbase2002.codfw.wmnet |
[production] |
| 14:04 |
<dcausse> |
restarting elastic on relforge100x servers to test new config |
[production] |
| 13:32 |
<moritzm> |
rebooting hydrogen for kernel update |
[production] |
| 13:15 |
<moritzm> |
rebooting achernar for kernel update |
[production] |
| 13:13 |
<hashar> |
Purged project-logos for eswikisource/eswikiquote high density logos T170604 : find static/images/project-logos -maxdepth 1 -type f| sed -e 's%^%https://en.wikipedia.org/%' |
[production] |
| 13:11 |
<hashar@tin> |
Synchronized wmf-config/InitialiseSettings.php: High density logos for es.wikisource - T170604 (duration: 00m 43s) |
[production] |
| 13:10 |
<hashar@tin> |
Synchronized static/images/project-logos: High density logos for es.wikisource - T170604 (duration: 00m 44s) |
[production] |
| 13:08 |
<hashar@tin> |
Synchronized wmf-config/InitialiseSettings.php: High density logos for es.wikiquote - T170604 (duration: 00m 49s) |
[production] |
| 13:07 |
<hashar@tin> |
Synchronized static/images/project-logos: High density logos for es.wikiquote - T170604 (duration: 00m 46s) |
[production] |
| 13:05 |
<dcausse> |
restarting elastic relforge100x servers to pick up new version of the ltr plugin |
[production] |
| 12:56 |
<elukey> |
rolling restart of aqs* for jvm upgrades |
[production] |
| 12:45 |
<moritzm> |
enabling mw1260 (jessie-based video scaler) for job processing |
[production] |
| 12:44 |
<gehel> |
restarting cassandra on maps clusters |
[production] |
| 12:37 |
<godog> |
powercycle ms-be1016, couldn't get getty output from console |
[production] |
| 12:09 |
<godog> |
upgrade diamond to 4.0.515 in eqiad - T97635 |
[production] |
| 12:08 |
<aude> |
ran rebuildTermSqlIndex.php on test.wikidata |
[production] |
| 11:59 |
<jynus> |
testing defragmenting pc2004 - if lag is created, ignore |
[production] |
| 11:58 |
<moritzm> |
installing binutils update from jessie point release |
[production] |
| 11:19 |
<marostegui> |
Start a run of "timeout 10h purgeParserCache.php" on terbium, which will be killed at around 21:00 UTC so it doesn't overlap with the normal cron run - T167784 |
[production] |
| 11:16 |
<moritzm> |
upgrading/restarting logstash* for openjdk security update |
[production] |
| 11:13 |
<marostegui> |
Killing old running instances of purgeParserCache.php in terbium - https://phabricator.wikimedia.org/T167784 |
[production] |
| 11:02 |
<moritzm> |
installing openjdk security updates on elastic* |
[production] |
| 10:50 |
<moritzm> |
installing openjdk security updates on restbase* |
[production] |
| 10:41 |
<marostegui> |
Run mwscript purgeParserCache.php --wiki=aawiki --age=1900800 --msleep 500 from terbium - T167784 |
[production] |
| 10:28 |
<marostegui@tin> |
Synchronized wmf-config/InitialiseSettings.php: Parsercache: Reduce expiration time to 22 days - T167784 (duration: 00m 44s) |
[production] |
| 10:27 |
<moritzm> |
upgrade restbase2010 to latest OpenJDK security update |
[production] |
| 09:20 |
<godog> |
upgrade diamond to 4.0.515 in codfw - T97635 |
[production] |
| 09:15 |
<moritzm> |
upgrade restbase-test* and restbase-dev* to latest OpenJDK security update |
[production] |
| 09:14 |
<godog> |
upgrade diamond to 4.0.515 in ulsfo and esams - T97635 |
[production] |
| 07:27 |
<moritzm> |
installing apache security updates on app servers in eqiad |
[production] |