|
2014-10-28
§
|
| 20:54 |
<hashar> |
Zuul deadlocked again. Restarting Gearman plugin on Jenkins |
[production] |
| 20:53 |
<awight> |
updated payments from 525988487d6bbd08ddad50badd88e34e34104292 to 647d1eb7d8cccb73fabf5ffded9f713d24576c37 |
[production] |
| 20:29 |
<manybubbles> |
removed /etc/elasticsearch/*.dpkg-dist fromg logstash machines - that was breaking logging for some reason. magic. |
[production] |
| 20:24 |
<manybubbles> |
disabling puppet on logstash1003 and trying to run elasticserach by hand to learn more about why its borked. |
[production] |
| 19:42 |
<andrewbogott> |
deleting unused labs projects: commons-dev, echo, farsi-wikitest |
[production] |
| 19:33 |
<bd808> |
Elasticsearch not recovering indices at all on logstash1003 and no logging output |
[production] |
| 19:11 |
<reedy> |
Synchronized php-1.25wmf5/extensions/CirrusSearch/: (no message) (duration: 00m 15s) |
[production] |
| 19:07 |
<Reedy> |
frwikibooks collation updated |
[production] |
| 19:06 |
<Reedy> |
Running mwscript updateCollation.php --wiki=frwikibooks --previous-collation=uppercase |
[production] |
| 19:05 |
<reedy> |
Synchronized wmf-config/: All of the config changes! (duration: 00m 14s) |
[production] |
| 18:53 |
<reedy> |
Synchronized wmf-config/: Bump cache epoch for Wikidata (duration: 00m 14s) |
[production] |
| 18:52 |
<reedy> |
Finished scap: Split Cite extension, scap to build l10n cache for CiteThisPage (duration: 33m 52s) |
[production] |
| 18:29 |
<bd808|LUNCH> |
restarted elasticsearch node on logstash1003 |
[production] |
| 18:18 |
<reedy> |
Started scap: Split Cite extension, scap to build l10n cache for CiteThisPage |
[production] |
| 18:11 |
<reedy> |
Synchronized wmf-config/: Set = true for Italian Wikipedia in November-December 2014 (duration: 00m 14s) |
[production] |
| 18:05 |
<reedy> |
rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.25wmf5 |
[production] |
| 17:27 |
<ori> |
Synchronized wmf-config/CommonSettings.php: I56082795: Modify $wmgAddWikiNotify for use by notifyNewProjects (duration: 00m 05s) |
[production] |
| 17:00 |
<bd808> |
restarted logstash on logstash1002 to try and get gelf input events into kibana again |
[production] |
| 16:58 |
<bd808> |
disk utilization on logstash100[123] greater than 80% |
[production] |
| 16:57 |
<cscott> |
no logs for ocg/parsoid on logstash since 2014-10-27T18:50:46.104Z/2014-10-27T18:50:45.977Z (respectively) |
[production] |
| 16:57 |
<andrew> |
Synchronized php-1.25wmf5/extensions/OpenStackManager: (no message) (duration: 00m 03s) |
[production] |
| 16:56 |
<bd808> |
No new logs in /var/log/elasticsearch for logstash100[123] since Sep 30 06:25 |
[production] |
| 16:55 |
<andrew> |
Synchronized php-1.25wmf5/extensions/OpenStackManager: (no message) (duration: 00m 02s) |
[production] |
| 16:44 |
<reedy> |
Synchronized php-1.25wmf4/extensions/OpenStackManager: (no message) (duration: 00m 14s) |
[production] |
| 15:47 |
<andrewbogott> |
running sync-common on virt1000 |
[production] |
| 15:47 |
<manybubbles> |
Synchronized php-1.25wmf4/extensions/OpenStackManager/: SWAT update openstackmanager (duration: 00m 04s) |
[production] |
| 15:46 |
<manybubbles> |
Synchronized php-1.25wmf5/extensions/OpenStackManager/: SWAT update openstackmanager (duration: 00m 04s) |
[production] |
| 15:24 |
<manybubbles> |
Synchronized wmf-config/: SWAT cirrus regex queues too small? (duration: 00m 05s) |
[production] |
| 15:11 |
<manybubbles> |
Synchronized php-1.25wmf5/extensions/Wikidata/: SWAT update wikidata (duration: 00m 10s) |
[production] |
| 15:03 |
<manybubbles> |
Synchronized wmf-config/: SWAT cirrus config updates - (hopefully) faster regexes (duration: 00m 06s) |
[production] |
| 14:51 |
<godog> |
rolling-restart of eqiad ms-fe* after https://gerrit.wikimedia.org/r/#/c/167310/ |
[production] |
| 14:04 |
<godog> |
reload swift frontend in eqiad after password rotation |
[production] |
| 14:04 |
<demon> |
Synchronized wmf-config/PrivateSettings.php: (no message) (duration: 00m 04s) |
[production] |
| 13:48 |
<manybubbles> |
Synchronized php-1.25wmf5/extensions/CirrusSearch/: (no message) (duration: 00m 05s) |
[production] |
| 13:47 |
<manybubbles> |
Synchronized php-1.25wmf4/extensions/CirrusSearch/: (no message) (duration: 00m 11s) |
[production] |
| 01:01 |
<demon> |
Synchronized wmf-config/InitialiseSettings.php: Turn Cirrus back on basically everywhere. If Elasticsearch freaks out again just revert I73ae276e to get back to lsearchd again (duration: 00m 04s) |
[production] |
| 00:43 |
<ori> |
Synchronized php-1.25wmf4/extensions/WikimediaEvents/WikimediaEventsHooks.php: I4adffaa26: Actually unset the HHVM cookie (duration: 00m 03s) |
[production] |
| 00:43 |
<ori> |
Synchronized php-1.25wmf5/extensions/WikimediaEvents/WikimediaEventsHooks.php: I4adffaa26: Actually unset the HHVM cookie (duration: 00m 03s) |
[production] |
| 00:27 |
<awight> |
reenabling recurring GlobalCollect job |
[production] |
| 00:07 |
<awight> |
updated crm from 9bb50403616d80aa8d39a89ab59965f53e9e3f3d to ffa543cab3eb508fa38b94c6de2643d168b0d507 |
[production] |
|
2014-10-27
§
|
| 23:52 |
<bd808> |
Restarted logstash service on logstash1001 because I was not seeing any events from MW make it into kibana |
[production] |
| 23:27 |
<maxsem> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/169229/ for reals now (duration: 00m 04s) |
[production] |
| 23:27 |
<Reedy> |
restarted logstash on logstash1001 |
[production] |
| 23:23 |
<maxsem> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/169229/ (duration: 00m 04s) |
[production] |
| 23:22 |
<Tim> |
on mw1114: disabled puppet, enabled Eval.PerfPidMap, restarted hhvm |
[production] |
| 23:21 |
<awight> |
updated crm from 5b395c37dc596736ecafceeb156221e3751bfe37 to 9bb50403616d80aa8d39a89ab59965f53e9e3f3d |
[production] |
| 23:21 |
<awight> |
disabling recurring globalcollect job |
[production] |
| 23:20 |
<maxsem> |
Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/168771/ (duration: 00m 04s) |
[production] |
| 23:17 |
<maxsem> |
Synchronized php-1.25wmf4/extensions/VisualEditor/: (no message) (duration: 00m 04s) |
[production] |
| 23:16 |
<maxsem> |
Synchronized php-1.25wmf4/extensions/MobileFrontend/: (no message) (duration: 00m 05s) |
[production] |