|
2016-11-10
§
|
| 14:47 |
<elukey> |
de-pooling mw1284 to raise mod_proxy_fcgi log level manually (temporary for an ongoing investigation) |
[production] |
| 14:31 |
<bblack> |
cache_text: upgrade nginx to 1.11.4-1+wmf14 |
[production] |
| 14:20 |
<bblack> |
cache_upload: upgrade nginx to 1.11.4-1+wmf14 |
[production] |
| 14:04 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1068 - T149079 (duration: 00m 48s) |
[production] |
| 13:54 |
<bblack> |
restarting varnishes on cache_maps + cache_misc |
[production] |
| 13:51 |
<marostegui> |
Enabled gtid+ssl on db2010,db2012,db2030 |
[production] |
| 13:45 |
<bblack> |
cache_misc: upgrade nginx to 1.11.4-1+wmf14 |
[production] |
| 13:29 |
<bblack> |
cache_maps: upgrade nginx to 1.11.4-1+wmf14 |
[production] |
| 13:25 |
<marostegui> |
Restarting mysql in misc shard slaves (only codfw - db2010,db2012,db2030) to apply a MySQL config - T149418 |
[production] |
| 13:20 |
<gehel> |
restart wdqs1* for jvm update |
[production] |
| 13:07 |
<gehel> |
restart wdqs2* for jvm update |
[production] |
| 12:01 |
<ema> |
upgrading cp1068 (text-eqiad) to varnish 4 -- T131503 |
[production] |
| 11:45 |
<ema> |
upgrading cp1067 (text-eqiad) to varnish 4 -- T131503 |
[production] |
| 11:25 |
<ema> |
upgrading cp1066 (text-eqiad) to varnish 4 -- T131503 |
[production] |
| 11:08 |
<ema> |
upgrading cp1065 (text-eqiad) to varnish 4 -- T131503 |
[production] |
| 10:43 |
<ema> |
upgrading cp1055 (text-eqiad) to varnish 4 -- T131503 |
[production] |
| 10:29 |
<ema> |
upgrading cp1054 (text-eqiad) to varnish 4 -- T131503 |
[production] |
| 10:26 |
<_joe_> |
powercycling mw1280, unresponsive to ping, blank unresponsive console |
[production] |
| 10:25 |
<moritzm> |
rolling restart of zookeeper in eqiad to pick up java security update |
[production] |
| 10:10 |
<moritzm> |
rolling restart of zookeeper in codfw to pick up java security update |
[production] |
| 09:53 |
<ema> |
upgrading cp1053 (text-eqiad) to varnish 4 -- T131503 |
[production] |
| 09:44 |
<marostegui> |
Deploy gtid_domain_id mysql flag for misc shards - https://phabricator.wikimedia.org/T149418 |
[production] |
| 09:43 |
<elukey> |
restarting druid daemons on druid100[123] for openjdk updates |
[production] |
| 09:38 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: wmf-config/db-codfw.php Depool db1059 - T149079. Repool db2048 T150334 (duration: 00m 50s) |
[production] |
| 09:22 |
<dcausse> |
elastic@codfw: reindexing commonswiki (logs in wasat.codfw.wmnet:~dcausse/commons_reindex/cirrus_log) |
[production] |
| 09:06 |
<moritzm> |
rolling restart of elasticsearch on logstash100[4-6] for picking up a Java security update |
[production] |
| 08:55 |
<moritzm> |
rebooting ruthenium for kernel update |
[production] |
| 08:30 |
<moritzm> |
rebooting copper for kernel update |
[production] |
| 08:06 |
<moritzm> |
rebooting bast3001 for kernel update |
[production] |
| 08:01 |
<marostegui> |
Deploy schema change s4 commonswiki.revision (dbstore1002) - T147305 |
[production] |
| 04:28 |
<mattflaschen@tin> |
Synchronized wmf-config/InitialiseSettings-labs.php: Beta Cluster only (duration: 00m 51s) |
[production] |
| 04:26 |
<mattflaschen@tin> |
Synchronized wmf-config/CommonSettings-labs.php: Beta Cluster only (duration: 00m 59s) |
[production] |
| 02:38 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Nov 10 02:38:24 UTC 2016 (duration 4m 36s) |
[production] |
| 02:33 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.2) (duration: 05m 52s) |
[production] |
| 02:19 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.1) (duration: 07m 23s) |
[production] |
| 00:46 |
<bblack> |
nginx-1.11.4-1+wmf14 uploaded to carbon jessie-wikimedia (only deployed to cp1008 for now) - T93927 - T148917 - T144523 |
[production] |
| 00:37 |
<godog> |
remove files on iridium:/tmp older than 5d - T150396 |
[production] |
| 00:14 |
<maxsem@tin> |
Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/313211/ (duration: 00m 52s) |
[production] |
| 00:09 |
<maxsem@tin> |
Synchronized wmf-config/CirrusSearch-labs.php: [labs only] https://gerrit.wikimedia.org/r/#/c/313037/ (duration: 00m 47s) |
[production] |
| 00:06 |
<maxsem@tin> |
Synchronized wmf-config/CirrusSearch-labs.php: [labs only] https://gerrit.wikimedia.org/r/#/c/313037/ (duration: 00m 48s) |
[production] |
|
2016-11-09
§
|
| 23:38 |
<godog> |
update cassandra aggregation scheme for 'count' metrics - T121789 |
[production] |
| 23:25 |
<godog> |
silence lutetium flapping check_mysql for two days |
[production] |
| 21:53 |
<eileen1> |
jobs stopped, dedupe (*2) donations queue |
[production] |
| 21:21 |
<eileen1> |
update CiviCRM from 7ee2ce47973c691e32d97ca6fed55df037cf585b to df50d2d7b6a8494fb3e049df6b1e101611270528 |
[production] |
| 21:18 |
<bearND> |
deployed mobileapps 106f4cd |
[production] |
| 21:14 |
<bearND> |
starting mobileapps deploy |
[production] |
| 20:59 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.2 |
[production] |
| 20:32 |
<urandom> |
T133395: Converting next 25 RESTBase tables to time-window compaction |
[production] |
| 20:13 |
<Krenair> |
created missing wikilove_log tables on azwiki and labtestwiki - T150321 |
[production] |
| 20:05 |
<Krinkle> |
Killed statsv.py process on hafnium. Seems to have fixed it. |
[production] |