|
2015-09-24
§
|
| 05:31 |
<yuvipanda> |
deleted indexes for 08/14, 15, 25, 26 on logstash |
[production] |
| 03:59 |
<yuvipanda> |
restarting elasticsearch in logstash1001-3 |
[production] |
| 03:53 |
<yuvipanda> |
restarted es on logstash1004-6 |
[production] |
| 03:02 |
<yuvipanda> |
jstack dumped logstash output onto /home/yuvipanda/stack on logstash1001 since strace seems useles |
[production] |
| 02:51 |
<yuvipanda> |
restarted logstash on logstash1002 |
[production] |
| 02:41 |
<yuvipanda> |
gmond at 100% again, killing it and stopping puppet again |
[production] |
| 02:40 |
<yuvipanda> |
re-enabling and running puppet on hafnium to see what it's bringing up |
[production] |
| 02:38 |
<l10nupdate@tin> |
Synchronized php-1.26wmf23/cache/l10n: l10nupdate for 1.26wmf23 (duration: 06m 30s) |
[production] |
| 02:23 |
<yuvipanda> |
kill gmond on hafnium and disable puppet to prevent it from taking it back up. Was taking 100% CPU |
[production] |
| 02:16 |
<Krinkle> |
Kibana/Logstash outage. Zero events received after 2015-09-23T23:59:59.999Z. |
[production] |
| 02:14 |
<Krinkle> |
Partial EventLogging outage (client-side events via hafnium abruptly stopped 2015-09-23 11:36 UTC - 15 hours ago) |
[production] |
| 01:53 |
<mutante> |
started logstash on logstash1002 again |
[production] |
| 01:35 |
<mutante> |
bast1001: unmounting /srv/home_pmtpa (backup on bacula) |
[production] |
| 01:34 |
<mutante> |
removing subversion packages from bast1001 |
[production] |
| 01:15 |
<ori@tin> |
Synchronized php-1.26wmf24/includes: Ifa0d4cfe8e3: Backport I1ff61153d and I8e4c3d5a5 (duration: 00m 23s) |
[production] |
| 00:19 |
<jynus> |
restarted replication on db1051 |
[production] |
| 00:17 |
<ori> |
restarting tcpircbot on neon |
[production] |
| 00:16 |
<mutante> |
started logstash on logstash1002 |
[production] |
| 00:08 |
<bblack> |
varnish package on carbon for jessie updated to 3.0.6plus-wm8 |
[production] |
|
2015-09-23
§
|
| 23:26 |
<thcipriani> |
Reloading Zuul to deploy Iaa33b2002e898d68fe78f236e225c6e4bac61679 |
[releng] |
| 23:17 |
<krenair@tin> |
Synchronized php-1.26wmf24/includes/specials/SpecialSearch.php: https://gerrit.wikimedia.org/r/#/c/240596/ (duration: 00m 18s) |
[production] |
| 23:09 |
<krenair@tin> |
Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/240575/ (duration: 00m 17s) |
[production] |
| 22:53 |
<ori@tin> |
Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: 14f46330d9: Backport fix from PS12 of I1ff6115 (duration: 00m 17s) |
[production] |
| 22:39 |
<paravoid> |
cr1-codfw RE switchover(s) |
[production] |
| 22:38 |
<thcipriani> |
Reloading Zuul to deploy I5db99dee0318b5854a05ea68870c57fb658f42e3 |
[releng] |
| 22:35 |
<yurik@tin> |
Synchronized php-1.26wmf24/extensions/ZeroBanner: Deploying ZeroBanner interstitial handling (duration: 00m 18s) |
[production] |
| 22:06 |
<ostriches> |
added tyler to gerrit admin group |
[releng] |
| 22:05 |
<marxarelli> |
Reloading zuul to deploy I55c30dd85444f84d21972b39a6237bab86518f13 |
[releng] |
| 22:05 |
<ostriches> |
added dan to gerrit admin group |
[releng] |
| 21:58 |
<hashar> |
Added thcipriani to the Gerrit 'integration' group |
[releng] |
| 21:58 |
<hashar> |
Added marxarelli and twentyafterfour (wm.org emails) to the Gerrit 'integration' group |
[releng] |
| 21:43 |
<marxarelli> |
Reloading zuul to deploy I02685b2f0f4215904e4a364e2f8597d1ab47d657 |
[releng] |
| 21:13 |
<ori@tin> |
Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: 58bfb6f85b: Backport fix from PS9 of I1ff6115 (duration: 00m 17s) |
[production] |
| 20:58 |
<marxarelli> |
Reloading zuul to deploy I4925da63b0aae22b3e809522466623d7958bfa24 |
[releng] |
| 20:46 |
<subbu> |
deployed parsoid 6619409e |
[production] |
| 20:23 |
<hashar> |
mwext-mw-selenium fails due to missing avconv https://phabricator.wikimedia.org/T113520 |
[releng] |
| 19:58 |
<urandom> |
starting Cassandra on restbase2006 |
[production] |
| 19:56 |
<urandom> |
enabling puppet, and forcing a run on restbase2006 |
[production] |
| 19:52 |
<urandom> |
starting Cassandra on restbase2005 |
[production] |
| 19:50 |
<urandom> |
enabling puppet, and forcing a run on restbase2005 |
[production] |
| 19:48 |
<urandom> |
starting Cassandra on restbase2004 |
[production] |
| 19:45 |
<urandom> |
enabling puppet, and forcing a run on restbase2004 |
[production] |
| 18:55 |
<twentyafterfour> |
snapshot1001.eqiad.wmnet returned [12]: rsync: write failed on "/srv/mediawiki/wikiversions.cdb": No space left on device |
[production] |
| 18:54 |
<twentyafterfour@tin> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf24 |
[production] |
| 18:36 |
<hashar> |
reapplying labels <tt>contintLabsSlave UbuntuTrusty phpflavor-hhvm</tt> on https://integration.wikimedia.org/ci/computer/integration-slave-trusty-1014/ . https://phabricator.wikimedia.org/T113489 |
[releng] |
| 18:22 |
<valhallasw`cloud> |
here = https://etherpad.wikimedia.org/p/74j8K2zIob |
[tools] |
| 18:22 |
<valhallasw`cloud> |
experimenting with https://github.com/jordansissel/fpm on tools-packages, and manually installing packages for that. Noting them here. |
[tools] |
| 18:01 |
<mobrovac> |
restbase deploying f65313ed |
[production] |
| 17:49 |
<urandom> |
starting Cassandra on restbase2003 |
[production] |
| 17:45 |
<demon@tin> |
Synchronized php-1.26wmf23/extensions/Wikidata: (no message) (duration: 00m 25s) |
[production] |