1451-1500 of 10000 results (12ms)
2014-11-27 §
17:55 <_joe_> restarted hhvm on mw1224, the alarm may have been lost in the puppet failure shower earlier [production]
17:24 <godog> removed /var/lib/carbon/whisper/archived/jenkins from tungsten [production]
17:09 <godog> upload txstatsd 0.7.0~bzr30-0ubuntu0+14 to precise-wikimedia on carbon [production]
16:49 <godog> upload missing txstatsd 1.0.0-1 _source package_ to carbon [production]
16:48 <godog> upload missing txstatsd 1.0.0-1 to carbon [production]
15:48 <hoo> Synchronized php-1.25wmf10/extensions/Wikidata/: Fixing a data model bug + enable Statements on Properties for testwikidata (duration: 00m 12s) [production]
15:34 <hoo> Synchronized wmf-config/Wikibase.php: Set "displayStatementsOnProperties" for wikidata/testwikidata (duration: 00m 06s) [production]
14:48 <akosiaris> upgrading librsvg throughout the fleet [production]
04:35 <springle> restarted squid3 on carbon, but glitches seem to be upstream [production]
04:23 <LocalisationUpdate> ResourceLoader cache refresh completed at Thu Nov 27 04:23:45 UTC 2014 (duration 23m 44s) [production]
03:45 <springle> puppet failures everywhere; transient apt timeout [production]
02:33 <LocalisationUpdate> completed (1.25wmf10) at 2014-11-27 02:33:36+00:00 [production]
02:33 <l10nupdate> Synchronized php-1.25wmf10/cache/l10n: (no message) (duration: 00m 01s) [production]
02:20 <LocalisationUpdate> completed (1.25wmf9) at 2014-11-27 02:20:52+00:00 [production]
02:20 <l10nupdate> Synchronized php-1.25wmf9/cache/l10n: (no message) (duration: 00m 02s) [production]
00:30 <catrope> Synchronized php-1.25wmf10/extensions/VisualEditor: SWAT (duration: 00m 06s) [production]
00:27 <ejegg> set TY batch size=400 [production]
00:09 <awight> disabled thank-you activity records [production]
00:09 <catrope> Synchronized php-1.25wmf10/extensions/VisualEditor: SWAT (duration: 00m 07s) [production]
00:09 <catrope> Synchronized php-1.25wmf9/extensions/VisualEditor: SWAT (duration: 00m 05s) [production]
00:07 <catrope> Synchronized wmf-config/InitialiseSettings.php: Re-enable VisualEditor on frwiktionary and svwiktionary (duration: 00m 06s) [production]
2014-11-26 §
23:57 <ejegg> set TY batch size=700 [production]
23:48 <YuviPanda> manually ran puppet merge on strontium, puppet merge on palladium didn't sync [production]
23:46 <ejegg> enabling TY mail send [production]
23:45 <ejegg> set TY batch size=1 [production]
23:44 <ejegg> updated crm from 96f66e6b6c947c4e4c32c4a4a32dc940dc3b1d60 to 68703898b7ebfb2a038f307f17788739114806e4 [production]
23:38 <hashar> Jenkins all happy after a restart. Crashing to bed [production]
22:56 <hashar> Killing Jenkins, it is deadlocked beyond repair [production]
22:46 <hashar> Jenkins still in deadlock, will hard restart Jenkins and Zuul soonish. [production]
22:38 <ejegg> disabled TY email sending [production]
22:36 <ejegg> enabled TY email sending [production]
22:34 <ejegg> enabled CiviMail record creation for TY emails [production]
22:24 <cscott> restarted ocg [production]
22:24 <gwicke> Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 05s) [production]
22:22 <hashar> Jenkins executors are in deadlock ( https://phabricator.wikimedia.org/T72597 ) [production]
22:17 <hashar> Bah there can only be one mediawiki-core-doxygen-publish job running, with all the merges that happened on mediawiki/core due to the release, there are currently six of them in the queue. They will all be processed eventually [production]
22:14 <hashar> mediawiki/core postmerge changes are stuck because mediawiki-core-doxygen-publish refuses to start. Attempted to retrigger them by promoting a change: <tt>gallium$ zuul promote --pipeline postmerge --changes 175960,1</tt> [production]
22:13 <ejegg> updated crm from d0a51250d2bdbf3c818ec0486af284691c7a61ff to 96f66e6b6c947c4e4c32c4a4a32dc940dc3b1d60 [production]
22:08 <hashar> investigating Zuul/Jenkins. Jenkins potentially has a deadlock [production]
22:02 <cscott> updated Parsoid to version 67e2596c [production]
21:53 <hashar> Restarting Gearman client. I am in a meeting, will cleanup later. [production]
21:33 <bd808> restarted logstash on logstash1001; log2udp events not being received [production]
21:22 <ejegg> disabled ty sending [production]
21:15 <hashar> Zuul stuck, restarting Gearman client [production]
21:01 <bd808> restarted elasticsearch on logstash1002 for OOM [production]
21:00 <bd808> restarted elasticsearch on logstash1003 for OOM [production]
20:57 <bd808> All three elasticsaerch nodes in the logstash clsuter think logstash1003 is master but ogstash-2014.11.26 is not allocated on any node [production]
20:56 <ejegg> enabled queue consumers [production]
20:51 <cscott> updated OCG to version 7d8f2b8bd496464041e3ef9c092732457cc8f7ef (did not restart ocg) [production]
20:50 <reedy> Synchronized php-1.25wmf10: (no message) (duration: 00m 47s) [production]