1351-1400 of 10000 results (28ms)
2014-11-14 §
15:56 <ottomata> upgrading analytics1024 to trusty [production]
15:46 <ottomata> analytics1003 (a cisco) is acting crazy, stuck in some loop while trying to boot. Am attempting to fix with power cycle [production]
14:35 <paravoid> cr1-ulsfo: setting up BGP with new transit provider [production]
09:16 <hashar> Zuul is back [production]
09:14 <hashar> Zuul is flapping [production]
07:38 <jgage> logstash hosts: elasticsearch moved to bigger disks [production]
04:36 <Tim> on mw1114 restarting hhvm [production]
04:17 <LocalisationUpdate> ResourceLoader cache refresh completed at Fri Nov 14 04:17:16 UTC 2014 (duration 17m 15s) [production]
04:03 <jgage> logstash1002 migration to new md0 complete [production]
02:30 <LocalisationUpdate> completed (1.25wmf8) at 2014-11-14 02:30:34+00:00 [production]
02:28 <Tim> progressively increasing load on mw1114, attempting to reproduce the previous overload [production]
02:19 <jgage> logstash1003 elasticsearch migration to new raid0 complete [production]
02:18 <LocalisationUpdate> completed (1.25wmf7) at 2014-11-14 02:18:01+00:00 [production]
01:39 <kaldari> Synchronized wmf-config/mobile.php: updating WikiGrok A/B test times (duration: 00m 03s) [production]
01:04 <kaldari> Synchronized php-1.25wmf7/extensions/MobileFrontend: (no message) (duration: 00m 05s) [production]
01:04 <kaldari> Synchronized php-1.25wmf7/extensions/WikiGrok: (no message) (duration: 00m 03s) [production]
00:49 <jgage> logstash1003: migrating elasticsearch data to new raid volume [production]
00:42 <kaldari> Synchronized wmf-config/mobile.php: Update WikiGrok A/B test times (duration: 00m 03s) [production]
00:20 <demon> Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 06s) [production]
00:15 <mutante> nickel - shutdown [production]
00:15 <demon> Synchronized php-1.25wmf8/extensions/VisualEditor: (no message) (duration: 00m 04s) [production]
00:14 <demon> Synchronized php-1.25wmf7/extensions/VisualEditor: (no message) (duration: 00m 05s) [production]
00:08 <demon> Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 04s) [production]
00:06 <demon> Synchronized wmf-config/: (no message) (duration: 00m 07s) [production]
00:04 <demon> Synchronized php-1.25wmf8/extensions/Echo: (no message) (duration: 00m 04s) [production]
00:04 <demon> Synchronized php-1.25wmf7/extensions/Echo: (no message) (duration: 00m 06s) [production]
2014-11-13 §
23:55 <mutante> nickel - remove from puppet,salt,icinga,stop services... [production]
23:52 <^d> restarted gitblit on antimony [production]
23:39 <kaldari> Synchronized wmf-config/mobile.php: Adding WikiGrok A/B test start and end times (duration: 00m 03s) [production]
22:19 <jgage> hadoop: analytics1010 is again active namenode [production]
22:14 <awight> Synchronized php-1.25wmf8/extensions/CentralNotice: push CentralNotice updates (duration: 00m 04s) [production]
22:13 <awight> Synchronized php-1.25wmf7/extensions/CentralNotice: push CentralNotice updates (duration: 00m 05s) [production]
22:12 <qchris> restarted EventLogging jobs that write to disk, to pick up config changes [production]
22:03 <jgage> failed over hadoop namenode to analytics1004 [production]
21:42 <awight> Synchronized wmf-config: Enabling CentralNotice banner choice on testwiki, take 2 (duration: 00m 06s) [production]
21:15 <cscott> updated Parsoid to version dabff010 [production]
20:51 <cmjohnson> powering down logstash1001 to add disks [production]
20:39 <cmjohnson> powering down logstash1002 to add disks [production]
20:37 <awight> CentralNotice noops deployed to all wikis [production]
20:36 <awight> Synchronized php-1.25wmf7/extensions/CentralNotice: push CentralNotice updates (duration: 00m 05s) [production]
20:33 <awight> Synchronized wmf-config: Enabling CentralNotice banner choice on testwiki (duration: 00m 04s) [production]
20:32 <bd808> Dropped replica count of all logstash indices except today to 0. Should make rolling restarts faster during hardware upgrade. [production]
20:25 <awight> Synchronized php-1.25wmf8/extensions/CentralNotice: push CentralNotice updates (duration: 00m 05s) [production]
20:19 <csteipp> patched bugs 71111 and 71394 in wmf7 and wmf8 [production]
20:14 <cmjohnson> powering down logstash1003 for a few mins to add disks [production]
19:52 <ottomata> starting upgrade to trusty on analytics1023 [production]
19:15 <awight> campaigns reenabled [production]
18:55 <awight> disabling CentralNotice campaigns [production]
17:49 <ottomata> preparing for trusty upgrade of analytics1003 [production]
16:57 <bd808> dropped replica count to 0 for logstash indices from 2014-10-30 and 2014-10-31. [production]