|
2014-10-09
§
|
| 11:57 |
<springle> |
xtrabackup db1016 to db2010 |
[production] |
| 11:39 |
<manybubbles> |
starting upgrade of elastic1009 |
[production] |
| 11:12 |
<_joe_> |
reenabled puppet on mw* |
[production] |
| 11:11 |
<godog> |
disabled puppet in ms-fe/ms-be in eqiad/codfw to merge container-sync changes |
[production] |
| 10:35 |
<_joe_> |
disabling puppet on most mw* hosts while testing apache changes |
[production] |
| 08:17 |
<_joe_> |
repooling mw102[3-5],mw1053 in the hhvm pool |
[production] |
| 07:15 |
<_joe_> |
reimaging mw102[3-5] to hhvm |
[production] |
| 07:02 |
<_joe_> |
reinstalling mw1053 |
[production] |
| 03:33 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Oct 9 03:33:47 UTC 2014 (duration 33m 46s) |
[production] |
| 02:30 |
<LocalisationUpdate> |
completed (1.25wmf2) at 2014-10-09 02:30:03+00:00 |
[production] |
| 02:18 |
<LocalisationUpdate> |
completed (1.25wmf1) at 2014-10-09 02:17:53+00:00 |
[production] |
|
2014-10-08
§
|
| 23:31 |
<mutante> |
importing xml dump to cawikimedia |
[production] |
| 23:28 |
<andrewbogott> |
running sync-common on virt1000 |
[production] |
| 23:27 |
<demon> |
Synchronized php-1.25wmf2/extensions/OpenStackManager: (no message) (duration: 00m 06s) |
[production] |
| 23:23 |
<demon> |
Synchronized php-1.25wmf2/extensions/Flow: (no message) (duration: 00m 05s) |
[production] |
| 23:14 |
<demon> |
Synchronized php-1.25wmf2/extensions/CommonsMetadata: (no message) (duration: 00m 06s) |
[production] |
| 22:40 |
<mutante> |
virt0 - deleted salt key, revoked puppet cert, removed from site.pp |
[production] |
| 22:39 |
<Reedy> |
Removed openjdk-6-* from logstash100[1-3] |
[production] |
| 22:07 |
<subbu> |
updated OCG to version def24eca |
[production] |
| 22:07 |
<mutante> |
tin - deleted empty pmtpa dsh group files |
[production] |
| 22:02 |
<mutante> |
tin - there are dozens of dsh groups that have been removed from repo long time ago but never got purged, but it isn't easy to tell what might still be used, so deleting all and letting puppet recreate might be risky? |
[production] |
| 20:48 |
<legoktm> |
currently running /home/legoktm/fixBug71749.php on terbium |
[production] |
| 19:49 |
<Reedy> |
logstash upgraded to 1.4.2-1 on logstash100[1-3] |
[production] |
| 19:46 |
<Reedy> |
Created flow tables on officewiki |
[production] |
| 17:04 |
<_joe_> |
load testing done |
[production] |
| 16:44 |
<_joe_> |
doing some load testing on the hhvm servers |
[production] |
| 16:09 |
<Reedy> |
elasticsearch upgraded on logstash1001 to 1.3.4 |
[production] |
| 16:08 |
<Reedy> |
elasticsearch upgraded on logstash100[23] to 1.3.4 |
[production] |
| 16:07 |
<Reedy> |
elasticsearch upgraded on logstash[ |
[production] |
| 15:08 |
<anomie> |
Synchronized php-1.25wmf2/extensions/VisualEditor/lib/ve/src/ce/ve.ce.Surface.js: SWAT: Revert "ve.ce.Surface: Magic workaround for broken Firefox cursoring" [[gerrit:164593]] (duration: 00m 09s) |
[production] |
| 15:04 |
<manybubbles> |
upgrading elastic1002 now |
[production] |
| 14:53 |
<_joe_> |
repooling mw1163 |
[production] |
| 14:20 |
<hasharBusy> |
disabled puppet on gallium to make sure a zuul config change stick in. {{gerrit|165481}} |
[production] |
| 14:19 |
<manybubbles> |
fixed missing elasticsearch extension jar file and brought elastic1001 back up. git fat betrayed us. |
[production] |
| 14:14 |
<hasharBusy> |
hard restarting zuul |
[production] |
| 14:03 |
<manybubbles> |
upgrading elastic1001 uncovered a bug in our highlighter that I have yet to diagnose. I removed that server from the rotation so we'll continue to use the old version. |
[production] |
| 12:44 |
<manybubbles> |
upgraded elastic1001 to Elasticsearch 1.3.2 -> 1.3.4, experimental highlighter 0.0.11 -> 0.0.12, and installed trigram accelerated regex search 0.0.1 |
[production] |
| 12:33 |
<manybubbles> |
deploying new elasticsearch plugins in preparation for minor Elasticsearch version upgrade today |
[production] |
| 11:02 |
<reedy> |
Synchronized docroot and w: good riddance to bad docroots (duration: 00m 16s) |
[production] |
| 09:27 |
<springle> |
Synchronized wmf-config/db-eqiad.php: isolate api traffic on s2 to db1054 and db1060 (duration: 01m 20s) |
[production] |
| 09:03 |
<springle> |
killed masses of sleeping connections on s2 slaves |
[production] |
| 08:11 |
<paravoid> |
powercycling rhenium, unresponsive |
[production] |
| 07:55 |
<springle> |
restart db2011 |
[production] |
| 04:31 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Oct 8 04:31:03 UTC 2014 (duration 31m 2s) |
[production] |
| 03:18 |
<LocalisationUpdate> |
completed (1.25wmf2) at 2014-10-08 03:18:44+00:00 |
[production] |
| 02:40 |
<LocalisationUpdate> |
completed (1.25wmf1) at 2014-10-08 02:40:48+00:00 |
[production] |
| 02:02 |
<tstarling> |
Finished scap: (no message) (duration: 09m 01s) |
[production] |
| 01:53 |
<tstarling> |
Started scap: (no message) |
[production] |
| 01:35 |
<tstarling> |
scap failed: CalledProcessError Command '('/usr/bin/git', 'rev-list', '-1', '@{upstream}')' returned non-zero exit status 128 (duration: 00m 14s) |
[production] |
| 01:35 |
<tstarling> |
Started scap: (no message) |
[production] |