1351-1400 of 10000 results (31ms)
2014-11-13 §
22:14 <awight> Synchronized php-1.25wmf8/extensions/CentralNotice: push CentralNotice updates (duration: 00m 04s) [production]
22:13 <awight> Synchronized php-1.25wmf7/extensions/CentralNotice: push CentralNotice updates (duration: 00m 05s) [production]
22:12 <qchris> restarted EventLogging jobs that write to disk, to pick up config changes [production]
22:03 <jgage> failed over hadoop namenode to analytics1004 [production]
21:42 <awight> Synchronized wmf-config: Enabling CentralNotice banner choice on testwiki, take 2 (duration: 00m 06s) [production]
21:15 <cscott> updated Parsoid to version dabff010 [production]
20:51 <cmjohnson> powering down logstash1001 to add disks [production]
20:39 <cmjohnson> powering down logstash1002 to add disks [production]
20:37 <awight> CentralNotice noops deployed to all wikis [production]
20:36 <awight> Synchronized php-1.25wmf7/extensions/CentralNotice: push CentralNotice updates (duration: 00m 05s) [production]
20:33 <awight> Synchronized wmf-config: Enabling CentralNotice banner choice on testwiki (duration: 00m 04s) [production]
20:32 <bd808> Dropped replica count of all logstash indices except today to 0. Should make rolling restarts faster during hardware upgrade. [production]
20:25 <awight> Synchronized php-1.25wmf8/extensions/CentralNotice: push CentralNotice updates (duration: 00m 05s) [production]
20:19 <csteipp> patched bugs 71111 and 71394 in wmf7 and wmf8 [production]
20:14 <cmjohnson> powering down logstash1003 for a few mins to add disks [production]
19:52 <ottomata> starting upgrade to trusty on analytics1023 [production]
19:15 <awight> campaigns reenabled [production]
18:55 <awight> disabling CentralNotice campaigns [production]
17:49 <ottomata> preparing for trusty upgrade of analytics1003 [production]
16:57 <bd808> dropped replica count to 0 for logstash indices from 2014-10-30 and 2014-10-31. [production]
16:49 <bd808> restarted elasticsearch on logstash1002 [production]
16:46 <bd808> dropped replica count to 0 for logstash indices from 2014-10-14 through 2014-10-29. See https://phabricator.wikimedia.org/P73 for the commands. [production]
16:45 <ottomata> preparing to upgrade analytics1026 to trusty [production]
16:21 <bd808> disk utilization is 94% on logstash1002, 92% on logstash1001 and 91% on logstash1003. Too much data in indices even with replica count bumped down to 1 for the small disks we have today. [production]
16:16 <bd808> logstash elasticsearch cluster is pretty messed up. logstash1002 has lost shards for all indices except for today, and it's master for that one. [production]
16:16 <manybubbles> Synchronized php-1.25wmf8/extensions/CirrusSearch/: SWAT update cirrussearch to fix slow prefix queries (duration: 00m 05s) [production]
16:14 <manybubbles> Synchronized wmf-config/CirrusSearch-production.php: SWAT reenable regex search now that it will not crash elasticsearch (duration: 00m 04s) [production]
16:13 <manybubbles> Synchronized wmf-config/CirrusSearch-common.php: SWAT reenable accelerated regex search (regex search still disabled) (duration: 00m 03s) [production]
16:11 <manybubbles> Synchronized wmf-config/InitialiseSettings.php: SWAT force summary when running checkuser query on all wikis (duration: 00m 04s) [production]
16:01 <manybubbles> Synchronized wmf-config/InitialiseSettings.php: SWAT revert JPG thumbnail chaining on all wikis except commons (duration: 00m 05s) [production]
15:27 <hashar:> deleted all content from https://doc.wikimedia.org/ :-( Will regenerate. [production]
15:09 <godog> rolling restart of object-auditor in swift codfw/eqiad to pick up changes [production]
15:06 <yurik> Synchronized php-1.25wmf8/extensions/ZeroPortal: updatidng ZeroPortal to master (duration: 01m 13s) [production]
15:04 <chasemp> phabricator upgrades T1203 [production]
14:43 <hashar:> restarted zuul-merger on gallium [production]
14:42 <hashar:> restarting Jenkins and Zuul [production]
12:45 <godog> investigating high iops on swift eqiad with paravoid, stopped object-auditor on ms-be1005 and ms-be1015 [production]
11:09 <hashar> resurrected morebots in #wikimedia-operations (see [[Morebots]]). [production]
11:08 <hashar> Killed Jenkins due to a deadlock [production]
11:08 <hashar> Killing Jenkins due to a deadlock [production]
02:52 <mutante> beta puppet freshness - UNKNOWN: No valid datapoints found .. since 13d [production]
02:30 <LocalisationUpdate> completed (1.25wmf8) at 2014-11-13 02:30:00+00:00 [production]
02:18 <LocalisationUpdate> completed (1.25wmf7) at 2014-11-13 02:18:44+00:00 [production]
00:46 <mutante> thulium - Could not intern from pson: expected value in object at '"[PHP]\\n\\n; puppet:t'! [production]
2014-11-12 §
21:59 <reedy> Synchronized wmf-config/: Set useLegacyUsageIndex = true for Wikibase client (duration: 00m 17s) [production]
20:57 <reedy> rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf8 [production]
20:55 <hashar> Restarting Jenkins, deadlock on deployment-bastion [production]
20:28 <reedy> Synchronized wmf-config/: (no message) (duration: 00m 15s) [production]
20:28 <reedy> Synchronized database lists: (no message) (duration: 00m 15s) [production]
20:25 <manybubbles> restarting elastic1021 to pick up new plugins [production]