|
2014-10-08
§
|
| 22:02 |
<mutante> |
tin - there are dozens of dsh groups that have been removed from repo long time ago but never got purged, but it isn't easy to tell what might still be used, so deleting all and letting puppet recreate might be risky? |
[production] |
| 20:48 |
<legoktm> |
currently running /home/legoktm/fixBug71749.php on terbium |
[production] |
| 19:49 |
<Reedy> |
logstash upgraded to 1.4.2-1 on logstash100[1-3] |
[production] |
| 19:46 |
<Reedy> |
Created flow tables on officewiki |
[production] |
| 17:04 |
<_joe_> |
load testing done |
[production] |
| 16:44 |
<_joe_> |
doing some load testing on the hhvm servers |
[production] |
| 16:09 |
<Reedy> |
elasticsearch upgraded on logstash1001 to 1.3.4 |
[production] |
| 16:08 |
<Reedy> |
elasticsearch upgraded on logstash100[23] to 1.3.4 |
[production] |
| 16:07 |
<Reedy> |
elasticsearch upgraded on logstash[ |
[production] |
| 15:08 |
<anomie> |
Synchronized php-1.25wmf2/extensions/VisualEditor/lib/ve/src/ce/ve.ce.Surface.js: SWAT: Revert "ve.ce.Surface: Magic workaround for broken Firefox cursoring" [[gerrit:164593]] (duration: 00m 09s) |
[production] |
| 15:04 |
<manybubbles> |
upgrading elastic1002 now |
[production] |
| 14:53 |
<_joe_> |
repooling mw1163 |
[production] |
| 14:20 |
<hasharBusy> |
disabled puppet on gallium to make sure a zuul config change stick in. {{gerrit|165481}} |
[production] |
| 14:19 |
<manybubbles> |
fixed missing elasticsearch extension jar file and brought elastic1001 back up. git fat betrayed us. |
[production] |
| 14:14 |
<hasharBusy> |
hard restarting zuul |
[production] |
| 14:03 |
<manybubbles> |
upgrading elastic1001 uncovered a bug in our highlighter that I have yet to diagnose. I removed that server from the rotation so we'll continue to use the old version. |
[production] |
| 12:44 |
<manybubbles> |
upgraded elastic1001 to Elasticsearch 1.3.2 -> 1.3.4, experimental highlighter 0.0.11 -> 0.0.12, and installed trigram accelerated regex search 0.0.1 |
[production] |
| 12:33 |
<manybubbles> |
deploying new elasticsearch plugins in preparation for minor Elasticsearch version upgrade today |
[production] |
| 11:02 |
<reedy> |
Synchronized docroot and w: good riddance to bad docroots (duration: 00m 16s) |
[production] |
| 09:27 |
<springle> |
Synchronized wmf-config/db-eqiad.php: isolate api traffic on s2 to db1054 and db1060 (duration: 01m 20s) |
[production] |
| 09:03 |
<springle> |
killed masses of sleeping connections on s2 slaves |
[production] |
| 08:11 |
<paravoid> |
powercycling rhenium, unresponsive |
[production] |
| 07:55 |
<springle> |
restart db2011 |
[production] |
| 04:31 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Oct 8 04:31:03 UTC 2014 (duration 31m 2s) |
[production] |
| 03:18 |
<LocalisationUpdate> |
completed (1.25wmf2) at 2014-10-08 03:18:44+00:00 |
[production] |
| 02:40 |
<LocalisationUpdate> |
completed (1.25wmf1) at 2014-10-08 02:40:48+00:00 |
[production] |
| 02:02 |
<tstarling> |
Finished scap: (no message) (duration: 09m 01s) |
[production] |
| 01:53 |
<tstarling> |
Started scap: (no message) |
[production] |
| 01:35 |
<tstarling> |
scap failed: CalledProcessError Command '('/usr/bin/git', 'rev-list', '-1', '@{upstream}')' returned non-zero exit status 128 (duration: 00m 14s) |
[production] |
| 01:35 |
<tstarling> |
Started scap: (no message) |
[production] |
| 01:32 |
<tstarling> |
scap failed: CalledProcessError Command '('/usr/bin/git', 'rev-list', '-1', '@{upstream}')' returned non-zero exit status 128 (duration: 00m 14s) |
[production] |
| 01:31 |
<tstarling> |
Started scap: (no message) |
[production] |
| 01:16 |
<tstarling> |
scap failed: CalledProcessError Command '('/usr/bin/git', 'rev-list', '-1', '@{upstream}')' returned non-zero exit status 128 (duration: 00m 25s) |
[production] |
| 01:16 |
<tstarling> |
Started scap: update for Wikidata crash bug |
[production] |
| 00:41 |
<mutante> |
searchidx1001 - same, fixed duplicate salt-minion |
[production] |
| 00:40 |
<mutante> |
osmium - salt-minion was running twice, stopped both, killed one, restarted properly |
[production] |
| 00:38 |
<mutante> |
cp3016 - why you report failed puppet unlike everyone else but then it works |
[production] |
| 00:34 |
<springle> |
long schema changes running from terbium. ok to kill osc_host.sh in emergency |
[production] |
| 00:01 |
<ori> |
Synchronized php-1.25wmf2/extensions/WikimediaEvents: Update WikimediaEvents for If9cdde0f0 (duration: 00m 03s) |
[production] |
| 00:01 |
<ori> |
Synchronized php-1.25wmf1/extensions/WikimediaEvents: Update WikimediaEvents for If9cdde0f0 (duration: 00m 04s) |
[production] |
|
2014-10-07
§
|
| 23:29 |
<andrewbogott> |
restarting every shutoff VM on virt1005 |
[production] |
| 23:20 |
<maxsem> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: https://gerrit.wikimedia.org/r/165393 (duration: 00m 04s) |
[production] |
| 22:54 |
<cscott> |
updated OCG to version c778ea8b898f8ad8c2b7ad9de78a75469e7ed061 |
[production] |
| 22:50 |
<mutante> |
db68,tarin - revoke the last remaining pmtpa certs |
[production] |
| 22:48 |
<ori> |
Synchronized php-1.25wmf1/extensions/WikimediaEvents: Update WikimediaEvents for Ied71b5032: Groundwork for HHVM productivity analysis (duration: 00m 04s) |
[production] |
| 22:47 |
<mutante> |
db60,db69-74,es4,es7,es10 - remove from icinga monitoring, puppet certs, salt keys |
[production] |
| 22:42 |
<ori> |
Synchronized php-1.25wmf2/extensions/WikimediaEvents: Update WikimediaEvents for Ied71b5032: Groundwork for HHVM productivity analysis (duration: 00m 04s) |
[production] |
| 22:40 |
<mutante> |
fenari - revoked puppet cert, rm salt key, rm from icinga ... |
[production] |
| 22:37 |
<andrewbogott> |
cycling power on virt1005 -- unresponsive |
[production] |
| 21:27 |
<mutante> |
mchenry - revoke puppet cert, clean storedconfigs/rm from icinga |
[production] |