|
2014-11-26
§
|
| 23:57 |
<ejegg> |
set TY batch size=700 |
[production] |
| 23:48 |
<YuviPanda> |
manually ran puppet merge on strontium, puppet merge on palladium didn't sync |
[production] |
| 23:46 |
<ejegg> |
enabling TY mail send |
[production] |
| 23:45 |
<ejegg> |
set TY batch size=1 |
[production] |
| 23:44 |
<ejegg> |
updated crm from 96f66e6b6c947c4e4c32c4a4a32dc940dc3b1d60 to 68703898b7ebfb2a038f307f17788739114806e4 |
[production] |
| 23:38 |
<hashar> |
Jenkins all happy after a restart. Crashing to bed |
[production] |
| 22:56 |
<hashar> |
Killing Jenkins, it is deadlocked beyond repair |
[production] |
| 22:46 |
<hashar> |
Jenkins still in deadlock, will hard restart Jenkins and Zuul soonish. |
[production] |
| 22:38 |
<ejegg> |
disabled TY email sending |
[production] |
| 22:36 |
<ejegg> |
enabled TY email sending |
[production] |
| 22:34 |
<ejegg> |
enabled CiviMail record creation for TY emails |
[production] |
| 22:24 |
<cscott> |
restarted ocg |
[production] |
| 22:24 |
<gwicke> |
Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 05s) |
[production] |
| 22:22 |
<hashar> |
Jenkins executors are in deadlock ( https://phabricator.wikimedia.org/T72597 ) |
[production] |
| 22:17 |
<hashar> |
Bah there can only be one mediawiki-core-doxygen-publish job running, with all the merges that happened on mediawiki/core due to the release, there are currently six of them in the queue. They will all be processed eventually |
[production] |
| 22:14 |
<hashar> |
mediawiki/core postmerge changes are stuck because mediawiki-core-doxygen-publish refuses to start. Attempted to retrigger them by promoting a change: <tt>gallium$ zuul promote --pipeline postmerge --changes 175960,1</tt> |
[production] |
| 22:13 |
<ejegg> |
updated crm from d0a51250d2bdbf3c818ec0486af284691c7a61ff to 96f66e6b6c947c4e4c32c4a4a32dc940dc3b1d60 |
[production] |
| 22:08 |
<hashar> |
investigating Zuul/Jenkins. Jenkins potentially has a deadlock |
[production] |
| 22:02 |
<cscott> |
updated Parsoid to version 67e2596c |
[production] |
| 21:53 |
<hashar> |
Restarting Gearman client. I am in a meeting, will cleanup later. |
[production] |
| 21:33 |
<bd808> |
restarted logstash on logstash1001; log2udp events not being received |
[production] |
| 21:22 |
<ejegg> |
disabled ty sending |
[production] |
| 21:15 |
<hashar> |
Zuul stuck, restarting Gearman client |
[production] |
| 21:01 |
<bd808> |
restarted elasticsearch on logstash1002 for OOM |
[production] |
| 21:00 |
<bd808> |
restarted elasticsearch on logstash1003 for OOM |
[production] |
| 20:57 |
<bd808> |
All three elasticsaerch nodes in the logstash clsuter think logstash1003 is master but ogstash-2014.11.26 is not allocated on any node |
[production] |
| 20:56 |
<ejegg> |
enabled queue consumers |
[production] |
| 20:51 |
<cscott> |
updated OCG to version 7d8f2b8bd496464041e3ef9c092732457cc8f7ef (did not restart ocg) |
[production] |
| 20:50 |
<reedy> |
Synchronized php-1.25wmf10: (no message) (duration: 00m 47s) |
[production] |
| 20:30 |
<reedy> |
Purged l10n cache for 1.25wmf6 |
[production] |
| 20:29 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf10 |
[production] |
| 20:28 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf9 |
[production] |
| 20:24 |
<reedy> |
Finished scap: testwiki to 1.25wmf10 and build l10n cache (duration: 49m 03s) |
[production] |
| 20:14 |
<ejegg> |
updated crm from e13cae8c418d29ef444899e0a70bbe03f4b7079d to d0a51250d2bdbf3c818ec0486af284691c7a61ff |
[production] |
| 20:13 |
<ejegg> |
disabling queue consumers |
[production] |
| 19:35 |
<reedy> |
Started scap: testwiki to 1.25wmf10 and build l10n cache |
[production] |
| 18:18 |
<ebernhardson> |
Synchronized wmf-config/InitialiseSettings.php: Whitelist converted lqt pages on officewiki (duration: 00m 07s) |
[production] |
| 16:54 |
<gwicke> |
Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 05s) |
[production] |
| 16:34 |
<Krinkle> |
Changed Jenkins default language from "en_US" to "en" ("Ignore browser settings" was already enabled). Not sure why, but it's back to English now. |
[production] |
| 16:16 |
<marktraceur> |
Synchronized wmf-config/: [SWAT] [config] 174793 Enable VisualEditor as a Beta Feature on most remaining wikis (duration: 00m 06s) |
[production] |
| 16:13 |
<Krinkle> |
Jenkins is displaying everything in French (both logged-in/logged-out users alike) |
[production] |
| 16:11 |
<marktraceur> |
Synchronized php-1.25wmf9/extensions/Flow/: [SWAT] [wmf9] 175941 "Provide user to local LQT api calls" for officewiki. (duration: 00m 08s) |
[production] |
| 12:25 |
<godog> |
stopped ocg on ocg1* |
[production] |
| 12:18 |
<godog> |
restarting ocg on ocg1001 |
[production] |
| 12:00 |
<godog> |
removing pdf files older than 14d from ocg100* |
[production] |
| 11:57 |
<godog> |
removing pdf files older than 14d from ocg1001 |
[production] |
| 06:48 |
<tstarling> |
Synchronized w/oauth-headers.php: (no message) (duration: 00m 05s) |
[production] |