|
2016-03-10
§
|
| 23:53 |
<urandom> |
Starting Cassandra cleanup op on restbase10{07,10,11}-{a,b}.eqiad.wmnet : T125842 |
[production] |
| 23:53 |
<urandom> |
Starting Cassandra cleanup op on restbase10{07,10,11}-{a,b}.eqiad.wmnet : T125832 |
[production] |
| 23:06 |
<twentyafterfour@tin> |
Synchronized php-1.27.0-wmf.16/languages/LanguageConverter.php: deploying https://gerrit.wikimedia.org/r/#/c/276467/ (duration: 00m 31s) |
[production] |
| 22:03 |
<csteipp> |
deployed patch for T129506 to wmf15 & 16 |
[production] |
| 21:55 |
<mutante> |
cygnus , killing akumar's processes |
[production] |
| 21:42 |
<twentyafterfour@tin> |
Synchronized php-1.27.0-wmf.16/extensions/Wikidata: Deploy https://gerrit.wikimedia.org/r/#/c/276473/ (duration: 02m 10s) |
[production] |
| 21:03 |
<twentyafterfour@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.16 |
[production] |
| 20:32 |
<cmjohnson1> |
replacing failed disk ms1001 |
[production] |
| 20:31 |
<AaronSchulz> |
Ran <<mwscriptwikiset extensions/WikimediaMaintenance/filebackend/setZoneAccess.php private.dblist --backend=local-multiwrite --private>> |
[production] |
| 19:23 |
<godog> |
enable puppet in cache_cluster in eqiad, followed by the rest |
[production] |
| 19:17 |
<godog> |
set timeline and math data containers as readable in swift codfw |
[production] |
| 18:53 |
<krinkle@tin> |
Synchronized wmf-config/db-codfw.php: I47954e21 (duration: 00m 30s) |
[production] |
| 18:46 |
<godog> |
sync wikipedia-commons-gwtoolset-metadata with swiftrepl eqiad -> codfw T129359 |
[production] |
| 18:43 |
<yurik> |
deployed latest kartotherian |
[production] |
| 18:19 |
<ori@tin> |
Synchronized php-1.27.0-wmf.15/includes/GlobalFunctions.php: wfShellExec() debug logging for T129467 (take 2) (duration: 00m 26s) |
[production] |
| 18:13 |
<jynus> |
importing zhwiki missing records from production to labs |
[production] |
| 18:13 |
<ori@tin> |
Synchronized php-1.27.0-wmf.15/includes/GlobalFunctions.php: wfShellExec() debug logging for T129467 (duration: 00m 28s) |
[production] |
| 17:44 |
<godog> |
running puppet in batches on cache_upload in eqiad after https://gerrit.wikimedia.org/r/#/c/276223/ |
[production] |
| 17:38 |
<filippo@tin> |
Synchronized wmf-config/filebackend-production.php: swift codfw sync replication T129089 (duration: 00m 39s) |
[production] |
| 17:19 |
<ori> |
cleared hhbc bytecode repo on mw1082 and mw1245 on suspicion that old translations were reused |
[production] |
| 16:36 |
<moritzm> |
mw1122: restarted hhvm |
[production] |
| 15:12 |
<akosiaris> |
restart hhvm on mw1107 |
[production] |
| 14:29 |
<moritzm> |
mw1107: restarted hhvm |
[production] |
| 14:15 |
<_joe_> |
restarted multiple times hhvm on mw112; it didn't help, so removed manually the warmup upstart task and the old sqlite cache; also disabling puppet on the machine |
[production] |
| 14:08 |
<urandom> |
increasing outbound stream throughput on restbase1002.eqiad.wmnet to 200mbps : T125842 |
[production] |
| 13:32 |
<bblack> |
restarted mw1122 hhvm |
[production] |
| 13:26 |
<bblack> |
mw1210: restarted hhvm |
[production] |
| 13:25 |
<moritzm> |
mw1241: restarted hhvm |
[production] |
| 13:20 |
<bblack> |
mw1188: restarted hhvm (before alert hit IRC, was already pending in icinga) |
[production] |
| 13:19 |
<bblack> |
mw1240: restarted hhvm |
[production] |
| 13:17 |
<bblack> |
mw1122: restarted hhvm, killed stale procs from Jan24 looking like: /usr/bin/hhvm --php -c /etc/hhvm/fcgi.ini -r echo ini_get("hhvm.jit_warmup_requests")?:11; |
[production] |
| 13:15 |
<bblack> |
restarting hhvm on mw1112 |
[production] |
| 13:12 |
<bblack> |
restarting hhvm on mw1248, mw1151 |
[production] |
| 13:06 |
<bblack> |
restarting hhvm on mw1251 ... |
[production] |
| 13:02 |
<bblack> |
restarting hhvm on mw1258 ... |
[production] |
| 13:01 |
<bblack> |
restarting hhvm on mw1243 |
[production] |
| 12:59 |
<bblack> |
restarting hhvm on mw1091 mw1249 mw1252 mw1242 ... |
[production] |
| 12:56 |
<bblack> |
restarting hhvm on mw1217 |
[production] |
| 12:01 |
<_joe_> |
restarted hhvm on mw1122, stuck at startup |
[production] |
| 10:39 |
<mobrovac> |
restbase deploy end of 26bd4aa28 |
[production] |
| 10:31 |
<jynus> |
setting up the rest of the cross-datacenter master-master connections pending in wmf databases |
[production] |
| 10:09 |
<godog> |
decommissioning restbase1002.eqiad.wmnet : T125842 |
[production] |
| 09:50 |
<volans@tin> |
Synchronized wmf-config/db-codfw.php: Rebalance external storage servers in codfw T127330 (duration: 00m 34s) |
[production] |
| 09:50 |
<mobrovac> |
restbase deploy start of 26bd4aa28 on restbase1001 |
[production] |
| 08:55 |
<ema> |
apache2 and hhvm restarted on mw1107, mw1122 and mw1119 |
[production] |
| 03:08 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Thu Mar 10 03:08:54 UTC 2016 (duration 9m 38s) |
[production] |
| 02:59 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.16) (duration: 12m 33s) |
[production] |
| 02:33 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.15) (duration: 14m 08s) |
[production] |
| 00:45 |
<catrope@tin> |
Synchronized php-1.27.0-wmf.15/extensions/ZeroBanner: SWAT (duration: 00m 28s) |
[production] |