201-250 of 10000 results (29ms)
2016-02-11 §
06:39 <_joe_> stipped swift on ms-be1008, disk full because object were being written to the root partition [production]
06:16 <bd808> Re-enabled puppet on logstash1003 and forced pupept run [production]
06:02 <bd808> Raised logstash mem limit to -Xms512m on logstash1003 [production]
05:57 <bd808> disabled puppet on logstash1003 to debug configuration [production]
05:40 <bd808> logstash process on logstash1003 flapping; continuing to investigate [production]
05:25 <bd808> Restarted logstash on logstash1003; killed by OOMKiler [production]
02:16 <ori> restarted HHVM on mw1130; locked up (T89912) [production]
02:15 <bd808> Fetched 1271d36 to tin:/srv/mediawiki-staging; should have been updated by sync-wikiversions [production]
01:58 <tgr@mira> rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.13 again [production]
01:47 <mutante> labservices1001 - out of disk [production]
01:46 <mutante> ms-be1008 - delete/gzip large syslog, out of disk [production]
01:41 <tgr@mira> Synchronized wmf-config/mobile.php: fix for T49647 (duration: 02m 15s) [production]
01:05 <krenair@mira> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/269883/1 (duration: 02m 11s) [production]
00:53 <krenair@mira> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/263051/ (duration: 02m 13s) [production]
00:48 <krenair@mira> Synchronized w/static/images/project-logos/wuuwiki.png: https://gerrit.wikimedia.org/r/#/c/263051/ (duration: 02m 13s) [production]
00:44 <krenair@mira> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/264937/ (duration: 02m 15s) [production]
00:35 <krenair@mira> Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/269827/1 (duration: 02m 13s) [production]
00:32 <krenair@mira> Synchronized w/static/images/project-logos/hiwikiquote.png: https://gerrit.wikimedia.org/r/#/c/269827/1 (duration: 02m 13s) [production]
00:12 <krenair@mira> Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/259439/ (duration: 02m 20s) [production]
2016-02-10 §
23:49 <tgr> switched mw1017 to wmf.13 (all groups) [production]
23:38 <ori@mira> rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to php-1.27.0-wmf.12 [production]
23:34 <ori> Restarted HHVM on mw1017 [production]
23:27 <urandom> performing rolling restart of Cassandra in restbase staging (experimental gc settings) [production]
23:15 <ori@mira> Synchronized wmf-config/mobile.php: I6946eccf9c: Better hack for T49647 (duration: 02m 19s) [production]
23:10 <ori@tin> Synchronized php-1.27.0-wmf.12/includes/WebResponse.php: I13fcc3ce4: Allow changing cookie options in WebResponseSetCookie hook (duration: 01m 30s) [production]
23:08 <ori@tin> Synchronized php-1.27.0-wmf.13/includes/WebResponse.php: I13fcc3ce4: Allow changing cookie options in WebResponseSetCookie hook (duration: 01m 37s) [production]
22:49 <krinkle@mira> Synchronized w/static.php: (no message) (duration: 02m 18s) [production]
22:40 <krenair@mira> Synchronized php-1.27.0-wmf.13/extensions/MobileFrontend/resources/mobile.loggingSchemas/SchemaEdit.js: https://gerrit.wikimedia.org/r/#/c/269755/ (duration: 02m 23s) [production]
22:36 <urandom> rolling restart of Cassandra staging complete (experimental gc settings) [production]
22:35 <krenair@mira> Synchronized php-1.27.0-wmf.12/extensions/MobileFrontend/resources/mobile.loggingSchemas/SchemaEdit.js: https://gerrit.wikimedia.org/r/#/c/269848/ (duration: 02m 18s) [production]
22:32 <yurik> deployed and restarted kartotherian services [production]
22:28 <urandom> performing rolling restart of Cassandra in restbase staging (experimental gc settings) [production]
22:23 <yurik> deployed and restarted tilerator & tileratorui services [production]
22:15 <ori> Restarted apache on palladium and strontium [production]
21:42 <urandom> ephemerally lowering compactor thread count on restbase1002 from 10 to 8 (limit combined working space) [production]
21:21 <subbu> finished deploying parsoid version 8976ab93 [production]
21:16 <hashar> CI dust have settled. Krinkle and I have pooled a lot more Trusty slaves to accommodate for the overload caused by switching to php55 (jobs run on Trusty) [production]
21:14 <subbu> synced code + restarted parsoid on wtp1001 as a canary [production]
21:10 <subbu> starting parsoid deploy [production]
20:24 <chasemp> tc per client shaping for labstore1001 test [production]
20:21 <demon@mira> Synchronized php-1.27.0-wmf.13/includes/interwiki/Interwiki.php: fix cache stuff (duration: 02m 18s) [production]
20:17 <apergos> cleaned up integrations slave trusty 1001,10012,10013, 1016, missed in first round. [production]
20:10 <demon@mira> rebuilt wikiversions.php and synchronized wikiversions files: rebuild [production]
20:10 <demon@mira> Synchronized multiversion/MWWikiversions.php: rm newline addition (duration: 02m 19s) [production]
20:03 <demon@mira> rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.13 [production]
19:45 <apergos> did cleanup across all integration slaves, some were very close to out of room. results: https://phabricator.wikimedia.org/P2587 [production]
19:26 <mutante> ms-be1008 - powercycling - the known XFS issue [production]
19:04 <mobrovac> restbase disabled puppet in staging, testing brotli compression which requires JAVA_OPTS tuning [production]
18:30 <mutante> puppetstoredconfigclean.rb iodine.wikimedia.org, revoke puppet cert, delete salt key on new master [production]
18:15 <mutante> iodine - shutdown, decom [production]