|
2015-09-24
§
|
| 11:56 |
<bblack> |
restarting varnish daemons on half of maps, parsoid, misc clusters (package upgrade, shm_reclen change) |
[production] |
| 11:36 |
<bblack> |
reinstall lvs300[12] to jessie - T96375 |
[production] |
| 11:21 |
<akosiaris> |
killed tail -f varnishncsa.log on cp1065 and ran apt-get clean to reclaim some disk space |
[production] |
| 11:14 |
<bblack> |
stopping pybal on lvs300[12]; lvs300[34] taking over |
[production] |
| 11:07 |
<bblack> |
upgrading varnishes to 3.0.6plus-wm8 (non-restarting, just pkg update on-disk) |
[production] |
| 09:40 |
<jynus> |
performing latest (software) steps to decom es1001-es1010 (puppet disabling, etc.) |
[production] |
| 08:39 |
<jynus> |
restarted HHVM @ mw1056, 1104, 1122 |
[production] |
| 06:09 |
<greg-g> |
deployment-bastion is getting close to having a filled up /var again: https://phabricator.wikimedia.org/T91354 |
[releng] |
| 05:33 |
<yuvipanda> |
deleted logstash indexes for 08/27 and 28 too |
[production] |
| 05:31 |
<yuvipanda> |
deleted indexes for 08/14, 15, 25, 26 on logstash |
[production] |
| 03:59 |
<yuvipanda> |
restarting elasticsearch in logstash1001-3 |
[production] |
| 03:53 |
<yuvipanda> |
restarted es on logstash1004-6 |
[production] |
| 03:02 |
<yuvipanda> |
jstack dumped logstash output onto /home/yuvipanda/stack on logstash1001 since strace seems useles |
[production] |
| 02:51 |
<yuvipanda> |
restarted logstash on logstash1002 |
[production] |
| 02:41 |
<yuvipanda> |
gmond at 100% again, killing it and stopping puppet again |
[production] |
| 02:40 |
<yuvipanda> |
re-enabling and running puppet on hafnium to see what it's bringing up |
[production] |
| 02:38 |
<l10nupdate@tin> |
Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 30s) |
[production] |
| 02:23 |
<yuvipanda> |
kill gmond on hafnium and disable puppet to prevent it from taking it back up. Was taking 100% CPU |
[production] |
| 02:16 |
<Krinkle> |
Kibana/Logstash outage. Zero events received after 2015-09-23T23:59:59.999Z. |
[production] |
| 02:14 |
<Krinkle> |
Partial EventLogging outage (client-side events via hafnium abruptly stopped 2015-09-23 11:36 UTC - 15 hours ago) |
[production] |
| 01:53 |
<mutante> |
started logstash on logstash1002 again |
[production] |
| 01:35 |
<mutante> |
bast1001: unmounting /srv/home_pmtpa (backup on bacula) |
[production] |
| 01:34 |
<mutante> |
removing subversion packages from bast1001 |
[production] |
| 01:15 |
<ori@tin> |
Synchronized php-1.26wmf24/includes: Ifa0d4cfe8e3: Backport I1ff61153d and I8e4c3d5a5 (duration: 00m 23s) |
[production] |
| 00:19 |
<jynus> |
restarted replication on db1051 |
[production] |
| 00:17 |
<ori> |
restarting tcpircbot on neon |
[production] |
| 00:16 |
<mutante> |
started logstash on logstash1002 |
[production] |
| 00:08 |
<bblack> |
varnish package on carbon for jessie updated to 3.0.6plus-wm8 |
[production] |
|
2015-09-23
§
|
| 23:26 |
<thcipriani> |
Reloading Zuul to deploy Iaa33b2002e898d68fe78f236e225c6e4bac61679 |
[releng] |
| 23:17 |
<krenair@tin> |
Synchronized php-1.26wmf24/includes/specials/SpecialSearch.php: https://gerrit.wikimedia.org/r/#/c/240596/ (duration: 00m 18s) |
[production] |
| 23:09 |
<krenair@tin> |
Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/240575/ (duration: 00m 17s) |
[production] |
| 22:53 |
<ori@tin> |
Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: 14f46330d9: Backport fix from PS12 of I1ff6115 (duration: 00m 17s) |
[production] |
| 22:39 |
<paravoid> |
cr1-codfw RE switchover(s) |
[production] |
| 22:38 |
<thcipriani> |
Reloading Zuul to deploy I5db99dee0318b5854a05ea68870c57fb658f42e3 |
[releng] |
| 22:35 |
<yurik@tin> |
Synchronized php-1.26wmf24/extensions/ZeroBanner: Deploying ZeroBanner interstitial handling (duration: 00m 18s) |
[production] |
| 22:06 |
<ostriches> |
added tyler to gerrit admin group |
[releng] |
| 22:05 |
<marxarelli> |
Reloading zuul to deploy I55c30dd85444f84d21972b39a6237bab86518f13 |
[releng] |
| 22:05 |
<ostriches> |
added dan to gerrit admin group |
[releng] |
| 21:58 |
<hashar> |
Added thcipriani to the Gerrit 'integration' group |
[releng] |
| 21:58 |
<hashar> |
Added marxarelli and twentyafterfour (wm.org emails) to the Gerrit 'integration' group |
[releng] |
| 21:43 |
<marxarelli> |
Reloading zuul to deploy I02685b2f0f4215904e4a364e2f8597d1ab47d657 |
[releng] |
| 21:13 |
<ori@tin> |
Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: 58bfb6f85b: Backport fix from PS9 of I1ff6115 (duration: 00m 17s) |
[production] |
| 20:58 |
<marxarelli> |
Reloading zuul to deploy I4925da63b0aae22b3e809522466623d7958bfa24 |
[releng] |
| 20:46 |
<subbu> |
deployed parsoid 6619409e |
[production] |
| 20:23 |
<hashar> |
mwext-mw-selenium fails due to missing avconv https://phabricator.wikimedia.org/T113520 |
[releng] |
| 19:58 |
<urandom> |
starting Cassandra on restbase2006 |
[production] |
| 19:56 |
<urandom> |
enabling puppet, and forcing a run on restbase2006 |
[production] |
| 19:52 |
<urandom> |
starting Cassandra on restbase2005 |
[production] |
| 19:50 |
<urandom> |
enabling puppet, and forcing a run on restbase2005 |
[production] |
| 19:48 |
<urandom> |
starting Cassandra on restbase2004 |
[production] |