|
2016-02-11
§
|
| 06:56 |
<_joe_> |
removing content from sd{d,j}1 written on the root partition on ms-be1008 |
[production] |
| 06:39 |
<_joe_> |
stipped swift on ms-be1008, disk full because object were being written to the root partition |
[production] |
| 06:16 |
<bd808> |
Re-enabled puppet on logstash1003 and forced pupept run |
[production] |
| 06:02 |
<bd808> |
Raised logstash mem limit to -Xms512m on logstash1003 |
[production] |
| 05:57 |
<bd808> |
disabled puppet on logstash1003 to debug configuration |
[production] |
| 05:40 |
<bd808> |
logstash process on logstash1003 flapping; continuing to investigate |
[production] |
| 05:25 |
<bd808> |
Restarted logstash on logstash1003; killed by OOMKiler |
[production] |
| 02:16 |
<ori> |
restarted HHVM on mw1130; locked up (T89912) |
[production] |
| 02:15 |
<bd808> |
Fetched 1271d36 to tin:/srv/mediawiki-staging; should have been updated by sync-wikiversions |
[production] |
| 01:58 |
<tgr@mira> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.13 again |
[production] |
| 01:47 |
<mutante> |
labservices1001 - out of disk |
[production] |
| 01:46 |
<mutante> |
ms-be1008 - delete/gzip large syslog, out of disk |
[production] |
| 01:41 |
<tgr@mira> |
Synchronized wmf-config/mobile.php: fix for T49647 (duration: 02m 15s) |
[production] |
| 01:05 |
<krenair@mira> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/269883/1 (duration: 02m 11s) |
[production] |
| 00:53 |
<krenair@mira> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/263051/ (duration: 02m 13s) |
[production] |
| 00:48 |
<krenair@mira> |
Synchronized w/static/images/project-logos/wuuwiki.png: https://gerrit.wikimedia.org/r/#/c/263051/ (duration: 02m 13s) |
[production] |
| 00:44 |
<krenair@mira> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264937/ (duration: 02m 15s) |
[production] |
| 00:35 |
<krenair@mira> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/269827/1 (duration: 02m 13s) |
[production] |
| 00:32 |
<krenair@mira> |
Synchronized w/static/images/project-logos/hiwikiquote.png: https://gerrit.wikimedia.org/r/#/c/269827/1 (duration: 02m 13s) |
[production] |
| 00:12 |
<krenair@mira> |
Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/259439/ (duration: 02m 20s) |
[production] |
|
2016-02-10
§
|
| 23:49 |
<tgr> |
switched mw1017 to wmf.13 (all groups) |
[production] |
| 23:38 |
<ori@mira> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to php-1.27.0-wmf.12 |
[production] |
| 23:34 |
<ori> |
Restarted HHVM on mw1017 |
[production] |
| 23:27 |
<urandom> |
performing rolling restart of Cassandra in restbase staging (experimental gc settings) |
[production] |
| 23:15 |
<ori@mira> |
Synchronized wmf-config/mobile.php: I6946eccf9c: Better hack for T49647 (duration: 02m 19s) |
[production] |
| 23:10 |
<ori@tin> |
Synchronized php-1.27.0-wmf.12/includes/WebResponse.php: I13fcc3ce4: Allow changing cookie options in WebResponseSetCookie hook (duration: 01m 30s) |
[production] |
| 23:08 |
<ori@tin> |
Synchronized php-1.27.0-wmf.13/includes/WebResponse.php: I13fcc3ce4: Allow changing cookie options in WebResponseSetCookie hook (duration: 01m 37s) |
[production] |
| 22:49 |
<krinkle@mira> |
Synchronized w/static.php: (no message) (duration: 02m 18s) |
[production] |
| 22:40 |
<krenair@mira> |
Synchronized php-1.27.0-wmf.13/extensions/MobileFrontend/resources/mobile.loggingSchemas/SchemaEdit.js: https://gerrit.wikimedia.org/r/#/c/269755/ (duration: 02m 23s) |
[production] |
| 22:36 |
<urandom> |
rolling restart of Cassandra staging complete (experimental gc settings) |
[production] |
| 22:35 |
<krenair@mira> |
Synchronized php-1.27.0-wmf.12/extensions/MobileFrontend/resources/mobile.loggingSchemas/SchemaEdit.js: https://gerrit.wikimedia.org/r/#/c/269848/ (duration: 02m 18s) |
[production] |
| 22:32 |
<yurik> |
deployed and restarted kartotherian services |
[production] |
| 22:28 |
<urandom> |
performing rolling restart of Cassandra in restbase staging (experimental gc settings) |
[production] |
| 22:23 |
<yurik> |
deployed and restarted tilerator & tileratorui services |
[production] |
| 22:15 |
<ori> |
Restarted apache on palladium and strontium |
[production] |
| 21:42 |
<urandom> |
ephemerally lowering compactor thread count on restbase1002 from 10 to 8 (limit combined working space) |
[production] |
| 21:21 |
<subbu> |
finished deploying parsoid version 8976ab93 |
[production] |
| 21:16 |
<hashar> |
CI dust have settled. Krinkle and I have pooled a lot more Trusty slaves to accommodate for the overload caused by switching to php55 (jobs run on Trusty) |
[production] |
| 21:14 |
<subbu> |
synced code + restarted parsoid on wtp1001 as a canary |
[production] |
| 21:10 |
<subbu> |
starting parsoid deploy |
[production] |
| 20:24 |
<chasemp> |
tc per client shaping for labstore1001 test |
[production] |
| 20:21 |
<demon@mira> |
Synchronized php-1.27.0-wmf.13/includes/interwiki/Interwiki.php: fix cache stuff (duration: 02m 18s) |
[production] |
| 20:17 |
<apergos> |
cleaned up integrations slave trusty 1001,10012,10013, 1016, missed in first round. |
[production] |
| 20:10 |
<demon@mira> |
rebuilt wikiversions.php and synchronized wikiversions files: rebuild |
[production] |
| 20:10 |
<demon@mira> |
Synchronized multiversion/MWWikiversions.php: rm newline addition (duration: 02m 19s) |
[production] |
| 20:03 |
<demon@mira> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.13 |
[production] |
| 19:45 |
<apergos> |
did cleanup across all integration slaves, some were very close to out of room. results: https://phabricator.wikimedia.org/P2587 |
[production] |
| 19:26 |
<mutante> |
ms-be1008 - powercycling - the known XFS issue |
[production] |
| 19:04 |
<mobrovac> |
restbase disabled puppet in staging, testing brotli compression which requires JAVA_OPTS tuning |
[production] |
| 18:30 |
<mutante> |
puppetstoredconfigclean.rb iodine.wikimedia.org, revoke puppet cert, delete salt key on new master |
[production] |