251-300 of 10000 results (13ms)
2014-10-29 §
13:11 <manybubbles> lowered redundancy on logstash from 3 way to 2 way [production]
13:01 <cmjohnson> powering down/replacing elastic1017 and elastic1018 [production]
12:59 <cmjohnson> disabling puppet on elastic1017 and 1018 [production]
12:01 <cmjohnson> elastic1001, elastic1008 and elastic1013 powering down to replace ssds RT7779 [production]
12:01 <springle> xtrabackup clone db1007 to db2029 [production]
07:16 <ori> repooled mw1189 w/patched hhvm (<https://phabricator.wikimedia.org/T820#16428>) [production]
03:39 <ori> upgraded mw1114 to custom package with patch from https://phabricator.wikimedia.org/T820#16428 applied [production]
2014-10-28 §
23:12 <demon> Synchronized php-1.25wmf5/extensions/Wikidata: (no message) (duration: 00m 10s) [production]
23:05 <_joe_> removed stale heap profile files from /run/hhvm on mw1114 [production]
23:02 <demon> Synchronized wmf-config/CommonSettings.php: extension distributor stuffs (duration: 00m 05s) [production]
22:09 <ejegg> updated crm from ffa543cab3eb508fa38b94c6de2643d168b0d507 to 1f0dc2ce0ab84765c085cc0ee369a7a047c0d005 [production]
21:05 <awight> reverted payments, from 647d1eb7d8cccb73fabf5ffded9f713d24576c37 to e3d235f881282120409e1a6ed1a3908ce9a63c26 [production]
21:02 <hashar> Zuul back in action. [production]
20:54 <hashar> Zuul deadlocked again. Restarting Gearman plugin on Jenkins [production]
20:53 <awight> updated payments from 525988487d6bbd08ddad50badd88e34e34104292 to 647d1eb7d8cccb73fabf5ffded9f713d24576c37 [production]
20:29 <manybubbles> removed /etc/elasticsearch/*.dpkg-dist fromg logstash machines - that was breaking logging for some reason. magic. [production]
20:24 <manybubbles> disabling puppet on logstash1003 and trying to run elasticserach by hand to learn more about why its borked. [production]
19:42 <andrewbogott> deleting unused labs projects: commons-dev, echo, farsi-wikitest [production]
19:33 <bd808> Elasticsearch not recovering indices at all on logstash1003 and no logging output [production]
19:11 <reedy> Synchronized php-1.25wmf5/extensions/CirrusSearch/: (no message) (duration: 00m 15s) [production]
19:07 <Reedy> frwikibooks collation updated [production]
19:06 <Reedy> Running mwscript updateCollation.php --wiki=frwikibooks --previous-collation=uppercase [production]
19:05 <reedy> Synchronized wmf-config/: All of the config changes! (duration: 00m 14s) [production]
18:53 <reedy> Synchronized wmf-config/: Bump cache epoch for Wikidata (duration: 00m 14s) [production]
18:52 <reedy> Finished scap: Split Cite extension, scap to build l10n cache for CiteThisPage (duration: 33m 52s) [production]
18:29 <bd808|LUNCH> restarted elasticsearch node on logstash1003 [production]
18:18 <reedy> Started scap: Split Cite extension, scap to build l10n cache for CiteThisPage [production]
18:11 <reedy> Synchronized wmf-config/: Set = true for Italian Wikipedia in November-December 2014 (duration: 00m 14s) [production]
18:05 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: Non wikipedias to 1.25wmf5 [production]
17:27 <ori> Synchronized wmf-config/CommonSettings.php: I56082795: Modify $wmgAddWikiNotify for use by notifyNewProjects (duration: 00m 05s) [production]
17:00 <bd808> restarted logstash on logstash1002 to try and get gelf input events into kibana again [production]
16:58 <bd808> disk utilization on logstash100[123] greater than 80% [production]
16:57 <cscott> no logs for ocg/parsoid on logstash since 2014-10-27T18:50:46.104Z/2014-10-27T18:50:45.977Z (respectively) [production]
16:57 <andrew> Synchronized php-1.25wmf5/extensions/OpenStackManager: (no message) (duration: 00m 03s) [production]
16:56 <bd808> No new logs in /var/log/elasticsearch for logstash100[123] since Sep 30 06:25 [production]
16:55 <andrew> Synchronized php-1.25wmf5/extensions/OpenStackManager: (no message) (duration: 00m 02s) [production]
16:44 <reedy> Synchronized php-1.25wmf4/extensions/OpenStackManager: (no message) (duration: 00m 14s) [production]
15:47 <andrewbogott> running sync-common on virt1000 [production]
15:47 <manybubbles> Synchronized php-1.25wmf4/extensions/OpenStackManager/: SWAT update openstackmanager (duration: 00m 04s) [production]
15:46 <manybubbles> Synchronized php-1.25wmf5/extensions/OpenStackManager/: SWAT update openstackmanager (duration: 00m 04s) [production]
15:24 <manybubbles> Synchronized wmf-config/: SWAT cirrus regex queues too small? (duration: 00m 05s) [production]
15:11 <manybubbles> Synchronized php-1.25wmf5/extensions/Wikidata/: SWAT update wikidata (duration: 00m 10s) [production]
15:03 <manybubbles> Synchronized wmf-config/: SWAT cirrus config updates - (hopefully) faster regexes (duration: 00m 06s) [production]
14:51 <godog> rolling-restart of eqiad ms-fe* after https://gerrit.wikimedia.org/r/#/c/167310/ [production]
14:04 <godog> reload swift frontend in eqiad after password rotation [production]
14:04 <demon> Synchronized wmf-config/PrivateSettings.php: (no message) (duration: 00m 04s) [production]
13:48 <manybubbles> Synchronized php-1.25wmf5/extensions/CirrusSearch/: (no message) (duration: 00m 05s) [production]
13:47 <manybubbles> Synchronized php-1.25wmf4/extensions/CirrusSearch/: (no message) (duration: 00m 11s) [production]
01:01 <demon> Synchronized wmf-config/InitialiseSettings.php: Turn Cirrus back on basically everywhere. If Elasticsearch freaks out again just revert I73ae276e to get back to lsearchd again (duration: 00m 04s) [production]
00:43 <ori> Synchronized php-1.25wmf4/extensions/WikimediaEvents/WikimediaEventsHooks.php: I4adffaa26: Actually unset the HHVM cookie (duration: 00m 03s) [production]