| 2014-11-18
      
      § | 
    
  | 14:55 | <springle> | fail over m2 to m2-slave (db1046); investigating db1020 | [production] | 
            
  | 14:44 | <hashar> | Gerrit web interface dead with: Cannot open ReviewDb {{bug|73555}} | [production] | 
            
  | 04:16 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Tue Nov 18 04:16:03 UTC 2014 (duration 16m 2s) | [production] | 
            
  | 02:26 | <LocalisationUpdate> | completed (1.25wmf8) at 2014-11-18 02:26:17+00:00 | [production] | 
            
  | 02:22 | <^d> | jenkins locale set from 'en' to 'en_US' since 'en' means Italian somehow. | [production] | 
            
  | 02:14 | <LocalisationUpdate> | completed (1.25wmf7) at 2014-11-18 02:14:18+00:00 | [production] | 
            
  | 01:27 | <Tim> | on osmium: removing ori's custom kernel and rebooting | [production] | 
            
  | 01:23 | <springle> | temporarily reassign db1004 for phab migration tests | [production] | 
            
  | 00:55 | <maxsem> | Synchronized php-1.25wmf8/resources/lib/oojs-ui/: https://gerrit.wikimedia.org/r/#/c/174029/ (duration: 00m 04s) | [production] | 
            
  | 00:53 | <bd808> | Restarted logstash on logstash1002; lots of errors in the log about GELF input >128 chunks | [production] | 
            
  | 00:50 | <bd808> | Restarted hung logstash process on logstash1001 | [production] | 
            
  | 00:47 | <maxsem> | Synchronized wmf-config: https://gerrit.wikimedia.org/r/173878 (duration: 00m 03s) | [production] | 
            
  | 00:44 | <maxsem> | Synchronized visualeditor.dblist: https://gerrit.wikimedia.org/r/172996 (duration: 00m 03s) | [production] | 
            
  | 00:43 | <maxsem> | Synchronized visualeditor-default.dblist: https://gerrit.wikimedia.org/r/172993 (duration: 00m 04s) | [production] | 
            
  | 00:34 | <maxsem> | Synchronized php-1.25wmf7/extensions/VisualEditor/: SWAT (duration: 00m 04s) | [production] | 
            
  | 00:33 | <maxsem> | Synchronized php-1.25wmf7/extensions/MobileFrontend/: SWAT (duration: 00m 04s) | [production] | 
            
  | 00:33 | <maxsem> | Synchronized php-1.25wmf7/extensions/WikiGrok: SWAT (duration: 00m 03s) | [production] | 
            
  | 00:32 | <maxsem> | Synchronized php-1.25wmf8/extensions/VisualEditor/: SWAT (duration: 00m 04s) | [production] | 
            
  | 00:24 | <maxsem> | Synchronized php-1.25wmf8/extensions/MobileFrontend/: SWAT (duration: 00m 04s) | [production] | 
            
  | 00:23 | <maxsem> | Synchronized php-1.25wmf8/extensions/WikiGrok/: (no message) (duration: 00m 04s) | [production] | 
            
  | 00:21 | <maxsem> | Synchronized php-1.25wmf8/extensions/VisualEditor/: SWAT (duration: 00m 04s) | [production] | 
            
  | 00:21 | <maxsem> | Synchronized php-1.25wmf8/extensions/Flow: SWAT (duration: 00m 05s) | [production] | 
            
  
    | 2014-11-17
      
      § | 
    
  | 23:59 | <awight> | Synchronized wmf-config: Enable new CentralNotice features on mediawikiwiki (duration: 00m 04s) | [production] | 
            
  | 23:35 | <awight> | Synchronized php-1.25wmf8/extensions/CentralNotice: push CentralNotice updates (duration: 00m 05s) | [production] | 
            
  | 22:50 | <cscott> | updated Parsoid to version 819b2cf4 | [production] | 
            
  | 22:03 | <awight> | Synchronized wmf-config: Enable new CentralNotice features on beta.wmflabs (duration: 00m 07s) | [production] | 
            
  | 20:48 | <ejegg> | Synchronized wmf-config: (no message) (duration: 00m 03s) | [production] | 
            
  | 20:39 | <ejegg> | Synchronized php-1.25wmf8/extensions/CentralNotice/: Update CentralNotice for client-side banner choice (duration: 00m 03s) | [production] | 
            
  | 20:02 | <ottomata> | starting upgrade of analytics1014 to trusty | [production] | 
            
  | 17:57 | <ottomata> | starting upgrade to trusty of analytics1013 (having trouble scheduling downtime in icinga right now) | [production] | 
            
  | 16:39 | <akosiaris> | upload etherpad-lite_1.4.1-1 on apt.wikimedia.org | [production] | 
            
  | 16:38 | <demon> | Synchronized php-1.25wmf8/extensions/SecurePoll/: (no message) (duration: 00m 05s) | [production] | 
            
  | 16:38 | <demon> | Synchronized php-1.25wmf7/extensions/SecurePoll/: (no message) (duration: 00m 05s) | [production] | 
            
  | 16:11 | <demon> | Synchronized php-1.25wmf8/extensions/CirrusSearch: (no message) (duration: 00m 04s) | [production] | 
            
  | 16:11 | <demon> | Synchronized php-1.25wmf7/extensions/CirrusSearch: (no message) (duration: 00m 05s) | [production] | 
            
  | 16:09 | <hashar> | Renamed job mediawiki-vendor-integration to mediawiki-phpunit {{bug|72787}} | [production] | 
            
  | 16:03 | <demon> | Synchronized wmf-config/CirrusSearch-common.php: more jobs (duration: 00m 04s) | [production] | 
            
  | 14:31 | <hashar> | Jenkins/Zuul: disconnected/reconnected Jenkins Gearman client | [production] | 
            
  | 13:38 | <apergos> | ran puppetstoredconfigclean.rb on db1017, it must have been missed in the rename | [production] | 
            
  | 12:17 | <akosiaris> | final reboot for xenon, cerium, praseodymium after a dist-upgrade -y | [production] | 
            
  | 11:27 | <ori> | Synchronized php-1.25wmf8/includes/Import.php: Icc19961fd: 'Debugging statements to try to diagnose bug 40009' (duration: 00m 08s) | [production] | 
            
  | 11:22 | <ori> | Synchronized wmf-config/InitialiseSettings.php: Ied0a7ab4b: Route Bug40009 logs to fluorine (duration: 00m 07s) | [production] | 
            
  | 09:51 | <_joe_> | restarting mw1187, all apache children stuck in apc_pthreadmutex_lock() | [production] | 
            
  | 09:10 | <akosiaris> | praseodymium reimaging | [production] | 
            
  | 08:53 | <oblivian> | Synchronized wmf-config/CommonSettings.php: Open HHVM to 25% of anons (duration: 00m 06s) | [production] | 
            
  | 08:50 | <hashar> | if there is any oddity with Jenkins/Zuul please poke me. I am on IRC all day today | [production] | 
            
  | 08:39 | <hashar> | Jenkins upgraded | [production] | 
            
  | 08:30 | <akosiaris> | reimaging cerium | [production] | 
            
  | 08:21 | <hashar> | Upgrading Jenkins | [production] | 
            
  | 04:15 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Mon Nov 17 04:15:03 UTC 2014 (duration 15m 2s) | [production] |