|
2014-10-08
§
|
| 14:19 |
<manybubbles> |
fixed missing elasticsearch extension jar file and brought elastic1001 back up. git fat betrayed us. |
[production] |
| 14:14 |
<hasharBusy> |
hard restarting zuul |
[production] |
| 14:03 |
<manybubbles> |
upgrading elastic1001 uncovered a bug in our highlighter that I have yet to diagnose. I removed that server from the rotation so we'll continue to use the old version. |
[production] |
| 12:44 |
<manybubbles> |
upgraded elastic1001 to Elasticsearch 1.3.2 -> 1.3.4, experimental highlighter 0.0.11 -> 0.0.12, and installed trigram accelerated regex search 0.0.1 |
[production] |
| 12:33 |
<manybubbles> |
deploying new elasticsearch plugins in preparation for minor Elasticsearch version upgrade today |
[production] |
| 11:02 |
<reedy> |
Synchronized docroot and w: good riddance to bad docroots (duration: 00m 16s) |
[production] |
| 09:27 |
<springle> |
Synchronized wmf-config/db-eqiad.php: isolate api traffic on s2 to db1054 and db1060 (duration: 01m 20s) |
[production] |
| 09:03 |
<springle> |
killed masses of sleeping connections on s2 slaves |
[production] |
| 08:11 |
<paravoid> |
powercycling rhenium, unresponsive |
[production] |
| 07:55 |
<springle> |
restart db2011 |
[production] |
| 04:31 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Wed Oct 8 04:31:03 UTC 2014 (duration 31m 2s) |
[production] |
| 03:18 |
<LocalisationUpdate> |
completed (1.25wmf2) at 2014-10-08 03:18:44+00:00 |
[production] |
| 02:40 |
<LocalisationUpdate> |
completed (1.25wmf1) at 2014-10-08 02:40:48+00:00 |
[production] |
| 02:02 |
<tstarling> |
Finished scap: (no message) (duration: 09m 01s) |
[production] |
| 01:53 |
<tstarling> |
Started scap: (no message) |
[production] |
| 01:35 |
<tstarling> |
scap failed: CalledProcessError Command '('/usr/bin/git', 'rev-list', '-1', '@{upstream}')' returned non-zero exit status 128 (duration: 00m 14s) |
[production] |
| 01:35 |
<tstarling> |
Started scap: (no message) |
[production] |
| 01:32 |
<tstarling> |
scap failed: CalledProcessError Command '('/usr/bin/git', 'rev-list', '-1', '@{upstream}')' returned non-zero exit status 128 (duration: 00m 14s) |
[production] |
| 01:31 |
<tstarling> |
Started scap: (no message) |
[production] |
| 01:16 |
<tstarling> |
scap failed: CalledProcessError Command '('/usr/bin/git', 'rev-list', '-1', '@{upstream}')' returned non-zero exit status 128 (duration: 00m 25s) |
[production] |
| 01:16 |
<tstarling> |
Started scap: update for Wikidata crash bug |
[production] |
| 00:41 |
<mutante> |
searchidx1001 - same, fixed duplicate salt-minion |
[production] |
| 00:40 |
<mutante> |
osmium - salt-minion was running twice, stopped both, killed one, restarted properly |
[production] |
| 00:38 |
<mutante> |
cp3016 - why you report failed puppet unlike everyone else but then it works |
[production] |
| 00:34 |
<springle> |
long schema changes running from terbium. ok to kill osc_host.sh in emergency |
[production] |
| 00:01 |
<ori> |
Synchronized php-1.25wmf2/extensions/WikimediaEvents: Update WikimediaEvents for If9cdde0f0 (duration: 00m 03s) |
[production] |
| 00:01 |
<ori> |
Synchronized php-1.25wmf1/extensions/WikimediaEvents: Update WikimediaEvents for If9cdde0f0 (duration: 00m 04s) |
[production] |
|
2014-10-07
§
|
| 23:29 |
<andrewbogott> |
restarting every shutoff VM on virt1005 |
[production] |
| 23:20 |
<maxsem> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: https://gerrit.wikimedia.org/r/165393 (duration: 00m 04s) |
[production] |
| 22:54 |
<cscott> |
updated OCG to version c778ea8b898f8ad8c2b7ad9de78a75469e7ed061 |
[production] |
| 22:50 |
<mutante> |
db68,tarin - revoke the last remaining pmtpa certs |
[production] |
| 22:48 |
<ori> |
Synchronized php-1.25wmf1/extensions/WikimediaEvents: Update WikimediaEvents for Ied71b5032: Groundwork for HHVM productivity analysis (duration: 00m 04s) |
[production] |
| 22:47 |
<mutante> |
db60,db69-74,es4,es7,es10 - remove from icinga monitoring, puppet certs, salt keys |
[production] |
| 22:42 |
<ori> |
Synchronized php-1.25wmf2/extensions/WikimediaEvents: Update WikimediaEvents for Ied71b5032: Groundwork for HHVM productivity analysis (duration: 00m 04s) |
[production] |
| 22:40 |
<mutante> |
fenari - revoked puppet cert, rm salt key, rm from icinga ... |
[production] |
| 22:37 |
<andrewbogott> |
cycling power on virt1005 -- unresponsive |
[production] |
| 21:27 |
<mutante> |
mchenry - revoke puppet cert, clean storedconfigs/rm from icinga |
[production] |
| 21:04 |
<mutante> |
dobson - revoke puppet cert, delete from storedconfigs/icinga, deleted from dsh |
[production] |
| 20:56 |
<K4-713> |
altered worldpay account settings for France on payments |
[production] |
| 20:48 |
<mutante> |
mexia - revoke salt,puppet,monitoring,storedconfigs |
[production] |
| 20:27 |
<mutante> |
pdf2/pdf3 - revoked puppet certs, removed from DNS & icinga |
[production] |
| 19:42 |
<mutante> |
temp. stopped icinga-wm |
[production] |
| 19:41 |
<mutante> |
restarting apache on palladium - mod_passenger fail |
[production] |
| 19:30 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 23s) |
[production] |
| 19:29 |
<reedy> |
Synchronized database lists: (no message) (duration: 00m 20s) |
[production] |
| 19:20 |
<Reedy> |
Created EducationProgram tables on cawiki |
[production] |
| 19:19 |
<reedy> |
Synchronized wmf-config/: (no message) (duration: 00m 26s) |
[production] |
| 19:09 |
<^d> |
cleared old files from runs on gallium tmpfs, testing should recover now. |
[production] |
| 18:45 |
<csteipp> |
deployed fix for bug 71749 |
[production] |
| 18:43 |
<mutante> |
sanger - deleted salt key, revoked puppet cert, rm icinga stored config, already out of DNS - Killing sanger.wikimedia.org...done. |
[production] |