|
2018-04-18
ยง
|
| 16:11 |
<ppchelko@tin> |
Finished deploy [cpjobqueue/deploy@749ae82]: Update dependencies and reduce dedupe logging rate (duration: 00m 43s) |
[production] |
| 16:10 |
<ppchelko@tin> |
Started deploy [cpjobqueue/deploy@749ae82]: Update dependencies and reduce dedupe logging rate |
[production] |
| 15:49 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2077 (duration: 01m 16s) |
[production] |
| 15:33 |
<_joe_> |
depooling mw1227 for investigation in high load |
[production] |
| 15:30 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1100 after alter table (duration: 01m 15s) |
[production] |
| 15:09 |
<urandom> |
decommissioning Cassandra, restbase1010-b -- T189822 |
[production] |
| 15:08 |
<dcausse> |
reindexing serbian wikis on elastic@eqiad (T189265) |
[production] |
| 14:55 |
<urandom> |
restarting Cassandra, restbase1011-a to test v 0.8 of Prometheus JMX exporter -- T192456 |
[production] |
| 14:51 |
<jynus> |
starting reimage of db2077 |
[production] |
| 14:37 |
<urandom> |
restarting Cassandra, restbase1011-a -- T192456 |
[production] |
| 14:34 |
<marostegui> |
Disable puppet on db1114 - T191996 |
[production] |
| 14:13 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2080, depool db2077 (duration: 01m 16s) |
[production] |
| 14:04 |
<gehel> |
powercycle unresponsive maps-test2001 |
[production] |
| 14:00 |
<elukey> |
restart kafka on kafka1001 and kafka2001 (jobqueues,eventbus) for opnejdk-7 upgrades |
[production] |
| 13:50 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1100 for alter table (duration: 01m 16s) |
[production] |
| 13:49 |
<marostegui> |
Deploy schema change on db1100 - T191519 T188299 T190148 |
[production] |
| 13:44 |
<moritzm> |
uploaded HHVM 3.18.5+dfsg-1+wmf7+icu57 to apt.wikimedia.org/jessie-wikimedia (includes a patch for the MEMC_VAL_COMPRESSION_ZLIB flag in the memcached module (T184854)) |
[production] |
| 13:20 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 after alter table (duration: 01m 15s) |
[production] |
| 13:17 |
<Amir1> |
EU SWAT is done |
[production] |
| 13:17 |
<moritzm> |
uploaded HHVM 3.18.5+dfsg-1+wmf7+deb9u1 to apt.wikimedia.org/stretch-wikimedia (includes a patch for the MEMC_VAL_COMPRESSION_ZLIB flag in the memcached module (T184854) |
[production] |
| 13:16 |
<ladsgroup@tin> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:427156|Limit page creation and edit rate on Wikidata (T184948)]] (duration: 01m 17s) |
[production] |
| 13:00 |
<jynus> |
starting reimage of db2080 |
[production] |
| 12:57 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2081, depool db2080 (duration: 01m 16s) |
[production] |
| 11:20 |
<vgutierrez> |
Repool (Re-enable BGP) lvs2004 - T191897 |
[production] |
| 11:02 |
<jynus> |
starting reimage of db2081 |
[production] |
| 10:57 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2081, repool db2082, es2013 (duration: 01m 15s) |
[production] |
| 10:45 |
<vgutierrez> |
Depool and reimage lvs2004 - T191897 |
[production] |
| 10:27 |
<vgutierrez> |
Repool (Re-enable BGP) in lvs2005 - T191897 |
[production] |
| 09:49 |
<hoo> |
Ran scap pull on mwdebug1001 after checking https://gerrit.wikimedia.org/r/427156 |
[production] |
| 09:49 |
<jynus> |
starting reimage of db2082 |
[production] |
| 09:46 |
<Amir1> |
start of deleting auto patrol actions in small wikis (T184485) |
[production] |
| 09:41 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2082 (duration: 01m 15s) |
[production] |
| 09:37 |
<moritzm> |
strip apache/nginx/nutcracker/hhvm from former image scaler (now spares) |
[production] |
| 09:32 |
<vgutierrez> |
Depool and reimage lvs2005 - T191897 |
[production] |
| 09:30 |
<marostegui> |
Deploy schema change on db1096:3315 - T191519 T188299 T190148 |
[production] |
| 09:30 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 for alter table (duration: 01m 22s) |
[production] |
| 09:27 |
<godog> |
reenable puppet fleetwide after https://gerrit.wikimedia.org/r/c/421860 |
[production] |
| 09:16 |
<moritzm> |
imported lz4 0.0~r131-2~wmf1+trusty1 for trusty-wikimedia to apt.wikimedia.org (needed to build HHVM 3.18 for trusty) |
[production] |
| 09:09 |
<godog> |
stop puppet agent fleetwide before applying https://gerrit.wikimedia.org/r/c/421860/ |
[production] |
| 09:08 |
<moritzm> |
reimaging mw1281 to stretch |
[production] |
| 09:04 |
<_joe_> |
restart HHVM on mw1223,mw1224, also repool them after investigation in crashes |
[production] |
| 08:59 |
<vgutierrez> |
Repool (Re-enable BGP) in lvs3003 - T191897 |
[production] |
| 08:44 |
<elukey> |
execute cumin 'analytics10[28-69]*' 'rm /etc/apt/preferences.d/r_* && apt-get update' to clear jessie backports apt config - T192348 |
[production] |
| 07:39 |
<vgutierrez> |
Depool and reimage lvs3003 as stretch - T191897 |
[production] |
| 06:49 |
<marostegui> |
Deploy schema change on s5 codfw master (db2052) this will generate lag in codfw - T191519 T188299 T190148 |
[production] |
| 06:43 |
<moritzm> |
installing ruby security updates for trusty |
[production] |
| 05:27 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1114 after changing RX buffers - T191996 (duration: 01m 09s) |
[production] |
| 05:20 |
<marostegui> |
Change RX buffers on db1114 - T191996 |
[production] |
| 05:20 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1114 (duration: 01m 15s) |
[production] |
| 05:13 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1092 after alter table (duration: 01m 16s) |
[production] |