|
2014-11-27
§
|
| 17:55 |
<_joe_> |
restarted hhvm on mw1224, the alarm may have been lost in the puppet failure shower earlier |
[production] |
| 17:24 |
<godog> |
removed /var/lib/carbon/whisper/archived/jenkins from tungsten |
[production] |
| 17:09 |
<godog> |
upload txstatsd 0.7.0~bzr30-0ubuntu0+14 to precise-wikimedia on carbon |
[production] |
| 16:49 |
<godog> |
upload missing txstatsd 1.0.0-1 _source package_ to carbon |
[production] |
| 16:48 |
<godog> |
upload missing txstatsd 1.0.0-1 to carbon |
[production] |
| 15:48 |
<hoo> |
Synchronized php-1.25wmf10/extensions/Wikidata/: Fixing a data model bug + enable Statements on Properties for testwikidata (duration: 00m 12s) |
[production] |
| 15:34 |
<hoo> |
Synchronized wmf-config/Wikibase.php: Set "displayStatementsOnProperties" for wikidata/testwikidata (duration: 00m 06s) |
[production] |
| 14:48 |
<akosiaris> |
upgrading librsvg throughout the fleet |
[production] |
| 04:35 |
<springle> |
restarted squid3 on carbon, but glitches seem to be upstream |
[production] |
| 04:23 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at Thu Nov 27 04:23:45 UTC 2014 (duration 23m 44s) |
[production] |
| 03:45 |
<springle> |
puppet failures everywhere; transient apt timeout |
[production] |
| 02:33 |
<LocalisationUpdate> |
completed (1.25wmf10) at 2014-11-27 02:33:36+00:00 |
[production] |
| 02:33 |
<l10nupdate> |
Synchronized php-1.25wmf10/cache/l10n: (no message) (duration: 00m 01s) |
[production] |
| 02:20 |
<LocalisationUpdate> |
completed (1.25wmf9) at 2014-11-27 02:20:52+00:00 |
[production] |
| 02:20 |
<l10nupdate> |
Synchronized php-1.25wmf9/cache/l10n: (no message) (duration: 00m 02s) |
[production] |
| 00:30 |
<catrope> |
Synchronized php-1.25wmf10/extensions/VisualEditor: SWAT (duration: 00m 06s) |
[production] |
| 00:27 |
<ejegg> |
set TY batch size=400 |
[production] |
| 00:09 |
<awight> |
disabled thank-you activity records |
[production] |
| 00:09 |
<catrope> |
Synchronized php-1.25wmf10/extensions/VisualEditor: SWAT (duration: 00m 07s) |
[production] |
| 00:09 |
<catrope> |
Synchronized php-1.25wmf9/extensions/VisualEditor: SWAT (duration: 00m 05s) |
[production] |
| 00:07 |
<catrope> |
Synchronized wmf-config/InitialiseSettings.php: Re-enable VisualEditor on frwiktionary and svwiktionary (duration: 00m 06s) |
[production] |
|
2014-11-26
§
|
| 23:57 |
<ejegg> |
set TY batch size=700 |
[production] |
| 23:48 |
<YuviPanda> |
manually ran puppet merge on strontium, puppet merge on palladium didn't sync |
[production] |
| 23:46 |
<ejegg> |
enabling TY mail send |
[production] |
| 23:45 |
<ejegg> |
set TY batch size=1 |
[production] |
| 23:44 |
<ejegg> |
updated crm from 96f66e6b6c947c4e4c32c4a4a32dc940dc3b1d60 to 68703898b7ebfb2a038f307f17788739114806e4 |
[production] |
| 23:38 |
<hashar> |
Jenkins all happy after a restart. Crashing to bed |
[production] |
| 22:56 |
<hashar> |
Killing Jenkins, it is deadlocked beyond repair |
[production] |
| 22:46 |
<hashar> |
Jenkins still in deadlock, will hard restart Jenkins and Zuul soonish. |
[production] |
| 22:38 |
<ejegg> |
disabled TY email sending |
[production] |
| 22:36 |
<ejegg> |
enabled TY email sending |
[production] |
| 22:34 |
<ejegg> |
enabled CiviMail record creation for TY emails |
[production] |
| 22:24 |
<cscott> |
restarted ocg |
[production] |
| 22:24 |
<gwicke> |
Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 05s) |
[production] |
| 22:22 |
<hashar> |
Jenkins executors are in deadlock ( https://phabricator.wikimedia.org/T72597 ) |
[production] |
| 22:17 |
<hashar> |
Bah there can only be one mediawiki-core-doxygen-publish job running, with all the merges that happened on mediawiki/core due to the release, there are currently six of them in the queue. They will all be processed eventually |
[production] |
| 22:14 |
<hashar> |
mediawiki/core postmerge changes are stuck because mediawiki-core-doxygen-publish refuses to start. Attempted to retrigger them by promoting a change: <tt>gallium$ zuul promote --pipeline postmerge --changes 175960,1</tt> |
[production] |
| 22:13 |
<ejegg> |
updated crm from d0a51250d2bdbf3c818ec0486af284691c7a61ff to 96f66e6b6c947c4e4c32c4a4a32dc940dc3b1d60 |
[production] |
| 22:08 |
<hashar> |
investigating Zuul/Jenkins. Jenkins potentially has a deadlock |
[production] |
| 22:02 |
<cscott> |
updated Parsoid to version 67e2596c |
[production] |
| 21:53 |
<hashar> |
Restarting Gearman client. I am in a meeting, will cleanup later. |
[production] |
| 21:33 |
<bd808> |
restarted logstash on logstash1001; log2udp events not being received |
[production] |
| 21:22 |
<ejegg> |
disabled ty sending |
[production] |
| 21:15 |
<hashar> |
Zuul stuck, restarting Gearman client |
[production] |
| 21:01 |
<bd808> |
restarted elasticsearch on logstash1002 for OOM |
[production] |
| 21:00 |
<bd808> |
restarted elasticsearch on logstash1003 for OOM |
[production] |
| 20:57 |
<bd808> |
All three elasticsaerch nodes in the logstash clsuter think logstash1003 is master but ogstash-2014.11.26 is not allocated on any node |
[production] |
| 20:56 |
<ejegg> |
enabled queue consumers |
[production] |