| 2014-12-04
      
      ยง | 
    
  | 22:36 | <awight> | Synchronized php-1.25wmf11/extensions/CentralNotice: push CentralNotice features (duration: 00m 06s) | [production] | 
            
  | 22:35 | <awight> | Synchronized php-1.25wmf10/extensions/CentralNotice: push CentralNotice features (duration: 00m 08s) | [production] | 
            
  | 22:01 | <YuviPanda> | nodejs sucks. | [production] | 
            
  | 21:36 | <awight> | Synchronized wmf-config: Shortening CentralNotice close box to 1 week (duration: 00m 07s) | [production] | 
            
  | 21:16 | <awight> | pushed CentralNotice fixes to hide from Google | [production] | 
            
  | 21:16 | <awight> | Synchronized php-1.25wmf11/extensions/CentralNotice: Hide CentralNotice banners from Google (duration: 00m 07s) | [production] | 
            
  | 21:16 | <awight> | Synchronized php-1.25wmf10/extensions/CentralNotice: Hide CentralNotice banners from Google (duration: 00m 06s) | [production] | 
            
  | 21:08 | <ori> | restarted ganglia-monitor on mw1081 | [production] | 
            
  | 20:56 | <legoktm> | removeInvalidEmails.php finished, removed a total of 218,598 emails | [production] | 
            
  | 20:17 | <awight> | Synchronized robots.txt: Disallow banner stuff in robots.txt (duration: 00m 07s) | [production] | 
            
  | 19:16 | <legoktm> | running removeInvalidEmails.php across all wikis | [production] | 
            
  | 18:57 | <demon> | Synchronized wmf-config/StartProfiler.php: (no message) (duration: 00m 05s) | [production] | 
            
  | 18:33 | <legoktm> | ran removeInvalidEmails.php on testwiki | [production] | 
            
  | 18:28 | <bd808> | Rolling restart of the elasticsearch cluster for logstash did not fix corrupted logstash-2014.11.30 index. It was worth a shot. | [production] | 
            
  | 18:23 | <bd808> | restarted elasticsearch on logstash1002 | [production] | 
            
  | 17:28 | <bd808> | restarted elasticsearch on logstash1003 to see if the missing indices would recover | [production] | 
            
  | 16:28 | <bd808> | restarted elasticsearch on logstash1001. | [production] | 
            
  | 14:40 | <godog> | upgrade to diamond 3.5 on trusty hosts in esams | [production] | 
            
  | 14:31 | <godog> | upgrade to diamond 3.5 on trusty hosts in esams | [production] | 
            
  | 14:14 | <hashar> | Jenkins: haven't had to restart it, I cancelled a few jobs and it went back up processing jobs.. | [production] | 
            
  | 14:12 | <hashar> | Jenkins in deadlock , restarting it ( https://phabricator.wikimedia.org/T72597 ) | [production] | 
            
  | 13:45 | <_joe_> | repooling mw1081 | [production] | 
            
  | 13:10 | <_joe_> | depooling mw1081 to activate hyperthreading | [production] | 
            
  | 12:09 | <paravoid> | powercycling rhenium, kernel locked up | [production] | 
            
  | 11:24 | <godog> | upgrade to diamond 3.5 on trusty hosts in ulsfo | [production] | 
            
  | 10:44 | <godog> | upgrade diamond to 3.5 on all trusty machines in codfw | [production] | 
            
  | 10:33 | <godog> | test-upgrade diamond 3.5 on swift in codfw | [production] | 
            
  | 10:01 | <ori> | disabling puppet on mw1081 and restarting hhvm with hhvm.server.stat_cache\t=true to observe impact | [production] | 
            
  | 09:47 | <godog> | upload diamond 3.5-1wmf1 to trusty-wikimedia | [production] | 
            
  | 08:12 | <akosiaris> | disable puppet on carbon. Playing with partman :) | [production] | 
            
  | 05:13 | <demon> | Synchronized wmf-config/InitialiseSettings.php: confirmedit disabled on closed wikis (duration: 00m 05s) | [production] | 
            
  | 04:23 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Thu Dec  4 04:23:26 UTC 2014 (duration 23m 25s) | [production] | 
            
  | 04:01 | <Tim> | on mw1189: restarting hhvm | [production] | 
            
  | 02:55 | <krinkle> | Synchronized w/: Ifbfb7dfd8fc0cd822b0 and I6594bc82b9de (duration: 00m 05s) | [production] | 
            
  | 02:32 | <LocalisationUpdate> | completed (1.25wmf11) at 2014-12-04 02:32:18+00:00 | [production] | 
            
  | 02:32 | <l10nupdate> | Synchronized php-1.25wmf11/cache/l10n: (no message) (duration: 00m 01s) | [production] | 
            
  | 02:20 | <LocalisationUpdate> | completed (1.25wmf10) at 2014-12-04 02:20:47+00:00 | [production] | 
            
  | 02:20 | <l10nupdate> | Synchronized php-1.25wmf10/cache/l10n: (no message) (duration: 00m 01s) | [production] | 
            
  | 02:09 | <springle> | labsdb1001 upgrade & reboot | [production] | 
            
  | 02:05 | <awight> | update crm from cd936bb433e9f107d860fb6e3da44c2ca2cb7742 to adfbbecbf949932932a3b6bc8c20c15e2a8054b2 | [production] | 
            
  | 01:57 | <springle> | labsdb1002 upgrade & reboot | [production] | 
            
  | 01:26 | <springle> | labsdb1003 upgrade & reboot | [production] | 
            
  | 00:40 | <maxsem> | Synchronized php-1.25wmf11/extensions/Wikidata/: (no message) (duration: 00m 13s) | [production] | 
            
  | 00:39 | <maxsem> | Synchronized php-1.25wmf10/extensions/Wikidata/: (no message) (duration: 00m 12s) | [production] | 
            
  | 00:25 | <maxsem> | Synchronized php-1.25wmf11/maintenance/removeInvalidEmails.php: https://gerrit.wikimedia.org/r/#/c/177021/ (duration: 00m 05s) | [production] | 
            
  | 00:25 | <maxsem> | Synchronized php-1.25wmf10/maintenance/removeInvalidEmails.php: https://gerrit.wikimedia.org/r/#/c/177021/ (duration: 00m 05s) | [production] | 
            
  | 00:21 | <maxsem> | Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/177271/ (duration: 00m 07s) | [production] | 
            
  | 00:11 | <maxsem> | Synchronized php-1.25wmf11/extensions/WikiGrok/: https://gerrit.wikimedia.org/r/177393 (duration: 00m 05s) | [production] | 
            
  | 00:10 | <maxsem> | Synchronized php-1.25wmf10/extensions/WikiGrok/: https://gerrit.wikimedia.org/r/177393 (duration: 00m 08s) | [production] | 
            
  | 00:04 | <maxsem> | Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/177153/ , noop in production (duration: 00m 07s) | [production] |