951-1000 of 10000 results (47ms)
2018-04-16 ยง
15:07 <jynus> start reimage of es3-codfw master, es2017 [production]
15:01 <vgutierrez@neodymium> conftool action : set/pooled=yes; selector: name=hydrogen.wikimedia.org,service=pdns_recursor [production]
14:53 <vgutierrez> restart pybal on lvs1003 - T187766 [production]
14:49 <vgutierrez> restart pybal on lvs2003 - T187766 [production]
14:43 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Fully repool db1114 in API - T191996 (duration: 00m 58s) [production]
14:42 <vgutierrez> restart pybal on lvs1006 - T187766 [production]
14:39 <gehel@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=wdqs-internal [production]
14:29 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Give more API traffic to db1114 (duration: 00m 57s) [production]
14:25 <vgutierrez> restarting pybal on lvs2006 - T187766 [production]
14:13 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Slowly repool db1114 in API (duration: 00m 58s) [production]
14:12 <moritzm> upgraded HHVM on mediawiki-deployment-09 to a build with a patch for the MEMC_VAL_COMPRESSION_ZLIB flag in the memcached module (T184854) [production]
14:06 <jynus> start reimage of es2-codfw master, es2016 [production]
14:05 <hashar> restarted Jenkins for plugin upgrade T192261 [production]
14:03 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore main traffic original weight for db1114 (duration: 00m 58s) [production]
13:47 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Slowly repool db1114 (duration: 00m 58s) [production]
13:32 <jynus@tin> Synchronized wmf-config/db-eqiad.php: Depool es1017 (duration: 00m 58s) [production]
13:31 <marostegui> Stop MySQL on db1114 to reboot with another kernel - T191996 [production]
13:30 <godog> roll-restart swift-proxy in codfw and eqiad - T188062 [production]
13:28 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1114 (duration: 00m 54s) [production]
13:18 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1087 (duration: 00m 59s) [production]
12:12 <vgutierrez@neodymium> conftool action : set/pooled=no; selector: name=hydrogen.wikimedia.org,service=pdns_recursor [production]
12:11 <vgutierrez> Depool and reimage hydrogen as stretch - T187090 [production]
11:50 <vgutierrez@neodymium> conftool action : set/pooled=yes; selector: name=acamar.wikimedia.org,service=pdns_recursor [production]
11:11 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore db1114 original weight (duration: 00m 59s) [production]
10:50 <moritzm> reimaging mw1299 (job runner) to stretch [production]
10:23 <ariel@tin> Finished deploy [dumps/dumps@4706d30]: show full stacktrace for dump job errors (duration: 00m 04s) [production]
10:23 <ariel@tin> Started deploy [dumps/dumps@4706d30]: show full stacktrace for dump job errors [production]
10:18 <godog> upload prometheus-memcached-exporter to stretch-wikimedia - T189056 [production]
10:17 <jdrewniak@tin> Synchronized portals: Wikimedia Portals Update: [[gerrit:426886|Bumping portals to master (T128546)]] (duration: 00m 58s) [production]
10:16 <jdrewniak@tin> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:426886|Bumping portals to master (T128546)]] (duration: 00m 59s) [production]
09:54 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Give more API traffic to db1114 (duration: 00m 58s) [production]
09:50 <vgutierrez@neodymium> conftool action : set/pooled=no; selector: name=acamar.wikimedia.org,service=pdns_recursor [production]
09:49 <vgutierrez> Depool and reimage acamar as stretch - T187090 [production]
09:43 <gehel> rolling restart of wdqs100[35] and wdqs200[123] for kernel upgrade completed [production]
09:40 <jynus> restarting dbstore2001:s8 to increase the number of purge threads [production]
09:23 <vgutierrez@neodymium> conftool action : set/pooled=yes; selector: name=achernar.wikimedia.org,service=pdns_recursor [production]
09:07 <gehel> starting rolling restart of wdqs100[35] and wdqs200[123] for kernel upgrade [production]
09:05 <moritzm> pooled mw1276-mw1278 (API app server canaries running stretch) [production]
08:49 <gehel> first manual run of populate_admin() for maps[12]001 - T190605 [production]
08:47 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore db1114 original main traffic weight (duration: 00m 58s) [production]
08:41 <moritzm> pooled mw1261-mw1264 (app server canaries running stretch) [production]
08:29 <joal@tin> Finished deploy [analytics/refinery@27416a9]: Regular weekly deploy - Mostly bugfixes from previous week huge deploy (duration: 05m 27s) [production]
08:25 <_joe_> depooling mw1223 for investigation too [production]
08:23 <joal@tin> Started deploy [analytics/refinery@27416a9]: Regular weekly deploy - Mostly bugfixes from previous week huge deploy [production]
08:17 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Slowly repool db1114 in API (duration: 00m 58s) [production]
08:04 <elukey> restart hhvm on mw[1228,1234,1281-1287,1289,1290,1312-1314,1317,1339,1343,1345,1346,1348] - more than 50% cpu usage, prevention scheme for current high load [production]
08:00 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Slowly repool db1114 (duration: 00m 58s) [production]
07:49 <marostegui> Stop MySQL and reboot db1114 - T191996 [production]
07:46 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1114 (duration: 00m 59s) [production]
07:40 <vgutierrez@neodymium> conftool action : set/pooled=no; selector: name=achernar.wikimedia.org,service=pdns_recursor [production]