|
2016-11-10
§
|
| 10:26 |
<_joe_> |
powercycling mw1280, unresponsive to ping, blank unresponsive console |
[production] |
| 10:25 |
<moritzm> |
rolling restart of zookeeper in eqiad to pick up java security update |
[production] |
| 10:10 |
<moritzm> |
rolling restart of zookeeper in codfw to pick up java security update |
[production] |
| 09:53 |
<ema> |
upgrading cp1053 (text-eqiad) to varnish 4 -- T131503 |
[production] |
| 09:44 |
<marostegui> |
Deploy gtid_domain_id mysql flag for misc shards - https://phabricator.wikimedia.org/T149418 |
[production] |
| 09:43 |
<elukey> |
restarting druid daemons on druid100[123] for openjdk updates |
[production] |
| 09:38 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: wmf-config/db-codfw.php Depool db1059 - T149079. Repool db2048 T150334 (duration: 00m 50s) |
[production] |
| 09:22 |
<dcausse> |
elastic@codfw: reindexing commonswiki (logs in wasat.codfw.wmnet:~dcausse/commons_reindex/cirrus_log) |
[production] |
| 09:06 |
<moritzm> |
rolling restart of elasticsearch on logstash100[4-6] for picking up a Java security update |
[production] |
| 08:55 |
<moritzm> |
rebooting ruthenium for kernel update |
[production] |
| 08:30 |
<moritzm> |
rebooting copper for kernel update |
[production] |
| 08:06 |
<moritzm> |
rebooting bast3001 for kernel update |
[production] |
| 08:01 |
<marostegui> |
Deploy schema change s4 commonswiki.revision (dbstore1002) - T147305 |
[production] |
| 04:28 |
<mattflaschen@tin> |
Synchronized wmf-config/InitialiseSettings-labs.php: Beta Cluster only (duration: 00m 51s) |
[production] |
| 04:26 |
<mattflaschen@tin> |
Synchronized wmf-config/CommonSettings-labs.php: Beta Cluster only (duration: 00m 59s) |
[production] |
| 02:38 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Nov 10 02:38:24 UTC 2016 (duration 4m 36s) |
[production] |
| 02:33 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.2) (duration: 05m 52s) |
[production] |
| 02:19 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.1) (duration: 07m 23s) |
[production] |
| 00:46 |
<bblack> |
nginx-1.11.4-1+wmf14 uploaded to carbon jessie-wikimedia (only deployed to cp1008 for now) - T93927 - T148917 - T144523 |
[production] |
| 00:37 |
<godog> |
remove files on iridium:/tmp older than 5d - T150396 |
[production] |
| 00:14 |
<maxsem@tin> |
Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/313211/ (duration: 00m 52s) |
[production] |
| 00:09 |
<maxsem@tin> |
Synchronized wmf-config/CirrusSearch-labs.php: [labs only] https://gerrit.wikimedia.org/r/#/c/313037/ (duration: 00m 47s) |
[production] |
| 00:06 |
<maxsem@tin> |
Synchronized wmf-config/CirrusSearch-labs.php: [labs only] https://gerrit.wikimedia.org/r/#/c/313037/ (duration: 00m 48s) |
[production] |
|
2016-11-09
§
|
| 23:38 |
<godog> |
update cassandra aggregation scheme for 'count' metrics - T121789 |
[production] |
| 23:25 |
<godog> |
silence lutetium flapping check_mysql for two days |
[production] |
| 21:53 |
<eileen1> |
jobs stopped, dedupe (*2) donations queue |
[production] |
| 21:21 |
<eileen1> |
update CiviCRM from 7ee2ce47973c691e32d97ca6fed55df037cf585b to df50d2d7b6a8494fb3e049df6b1e101611270528 |
[production] |
| 21:18 |
<bearND> |
deployed mobileapps 106f4cd |
[production] |
| 21:14 |
<bearND> |
starting mobileapps deploy |
[production] |
| 20:59 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.2 |
[production] |
| 20:32 |
<urandom> |
T133395: Converting next 25 RESTBase tables to time-window compaction |
[production] |
| 20:13 |
<Krenair> |
created missing wikilove_log tables on azwiki and labtestwiki - T150321 |
[production] |
| 20:05 |
<Krinkle> |
Killed statsv.py process on hafnium. Seems to have fixed it. |
[production] |
| 20:05 |
<Krinkle> |
statsv->graphite has been down for 9 hours since roughly 10AM UTC |
[production] |
| 19:51 |
<dereckson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable <mapframe> on ruwiki (T138057) (duration: 00m 48s) |
[production] |
| 19:12 |
<godog> |
upload cassandra-tools-wmf 1.0.0-1 to jessie-wikimedia on carbon - T150304 |
[production] |
| 18:47 |
<marostegui> |
Stopping MySQL dbstore2001 - taking a snapshot - T149457 |
[production] |
| 18:37 |
<jynus> |
partitioning db2042- it will have temporarily lag for 10-20 hours |
[production] |
| 18:36 |
<mutante> |
neon stopping nsca and apache |
[production] |
| 18:33 |
<mutante> |
neon (formerly icinga) remove from puppet, revoke cert, delete salt key, stop icinga service ... |
[production] |
| 18:33 |
<godog> |
deploy python-thumbor-wikimedia 0.1.29 to thumbor100[12] |
[production] |
| 17:41 |
<robh> |
the puppet failures on the frack hosts are known and have been reported to jeff |
[production] |
| 17:34 |
<jynus> |
rebooting db2048 for kernel upgrade |
[production] |
| 16:18 |
<ema> |
upgrading cp3043 (text-esams) to varnish 4 -- T131503 |
[production] |
| 15:58 |
<ema> |
upgrading cp3042 (text-esams) to varnish 4 -- T131503 |
[production] |
| 15:47 |
<urandom> |
T133395: Converting the next 25 RESTBase keyspaces to TWCS |
[production] |
| 15:34 |
<ema> |
upgrading cp3041 (text-esams) to varnish 4 -- T131503 |
[production] |
| 15:30 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2048 (duration: 00m 50s) |
[production] |
| 15:23 |
<ema> |
upgrading cp3040 (text-esams) to varnish 4 -- T131503 |
[production] |
| 15:11 |
<ema> |
upgrading cp3033 (text-esams) to varnish 4 -- T131503 |
[production] |