| 2015-09-28
      
      § | 
    
  | 20:26 | <yuvipanda> | depooled cp2017 from pybal config too | [production] | 
            
  | 20:22 | <yuvipanda> | depooled cp2017 since it's down | [production] | 
            
  | 19:02 | <chasemp> | powercycle iridium via console as it's unresponsive | [production] | 
            
  | 15:50 | <krenair@tin> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241579/ (duration: 00m 17s) | [production] | 
            
  | 15:28 | <krenair@tin> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241598/ (duration: 00m 17s) | [production] | 
            
  | 15:11 | <krenair@tin> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241649/ (duration: 00m 17s) | [production] | 
            
  | 14:00 | <paravoid> | running failover eqiad->codfw test for all frontend traffic | [production] | 
            
  | 06:11 | <ori@tin> | Synchronized wmf-config/CommonSettings.php: I179be4bd3: Rely on timeouts specified in php.ini rather than calling ini_set() (duration: 00m 17s) | [production] | 
            
  | 05:52 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Mon Sep 28 05:52:34 UTC 2015 (duration 52m 33s) | [production] | 
            
  | 02:27 | <l10nupdate@tin> | LocalisationUpdate completed (1.26wmf24) at 2015-09-28 02:27:16+00:00 | [production] | 
            
  | 02:20 | <l10nupdate@tin> | Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 06m 54s) | [production] | 
            
  
    | 2015-09-25
      
      § | 
    
  | 23:54 | <ori@tin> | Synchronized php-1.26wmf24/includes/GlobalFunctions.php: 37c6972f94: Made wfIsBadImage() use APC (duration: 00m 17s) | [production] | 
            
  | 22:16 | <mutante> | sodium - shutting down | [production] | 
            
  | 21:53 | <ori> | Armed new servicedeploy_rsa on mira | [production] | 
            
  | 21:51 | <ori> | Armed new servicedeploy_rsa on tin | [production] | 
            
  | 20:45 | <urandom> | starting a Cassandra repair on xeon (nodetool repair -pr) | [production] | 
            
  | 19:30 | <mutante> | restarted apache on gallium | [production] | 
            
  | 19:07 | <bblack> | configuring and enabling lvs-cross-row ports on asw-b-eqiad for lvs1007,8,10,11 | [production] | 
            
  | 18:04 | <jynus> | performing schema change on m5-master "nova" | [production] | 
            
  | 17:26 | <jynus> | restarting and upgrading db1051 mysql (depooled) | [production] | 
            
  | 17:20 | <godog> | reboot ms-be1005, xfs | [production] | 
            
  | 16:17 | <akosiaris> | duplicating database otrs to otrsupgradetest for testing the upgrade procedure | [production] | 
            
  | 16:16 | <cmjohnson1> | cp1046 stopped icinga checks for hardware troubleshooting | [production] | 
            
  | 14:43 | <moritzm> | installed rpcbind and apport security updates on various servers | [production] | 
            
  | 14:36 | <krenair@tin> | Synchronized php-1.26wmf24/includes/specials/SpecialMovepage.php: https://gerrit.wikimedia.org/r/#/c/241045/ (duration: 00m 17s) | [production] | 
            
  | 14:22 | <chasemp> | 'sudo -u hdfs hdfs haadmin -transitionToActive analytics1001-eqiad-wmnet' per otto on analytics1001 | [production] | 
            
  | 14:03 | <urandom> | starting a Cassandra repair on restbase1006 (nodetool repair -pr -dc eqiad) | [production] | 
            
  | 13:58 | <hashar> | nodepool back in operations | [production] | 
            
  | 13:44 | <moritzm> | restarted saltmaster on palladium | [production] | 
            
  | 13:12 | <mobrovac> | restbase deploying e42bf0fc | [production] | 
            
  | 13:12 | <moritzm> | added debdeploy 0.0.7 to carbon | [production] | 
            
  | 13:10 | <godog> | enable puppet on restbase in production | [production] | 
            
  | 12:54 | <andrewbogott> | restarted rabbitmq-server on labcontrol1001 | [production] | 
            
  | 12:52 | <godog> | stop puppet on restbase in production -- config deployment | [production] | 
            
  | 12:46 | <hashar> | stopping nodepool to clear out left over mysql connections | [production] | 
            
  | 12:21 | <hashar> | Nodepool is dead, or at least not adding new slaves anymore | [production] | 
            
  | 12:16 | <hashar> | Seems Zuul/Jenkins is in trouble somehow :-/ | [production] | 
            
  | 11:42 | <andrewbogott> | upgrading linux-image-generic on labnet1002 to get us away from the recently-crashed 3.13.0-59 | [production] | 
            
  | 04:58 | <ori@tin> | Synchronized wmf-config/CommonSettings.php: I252970886: Set "async" for SQL parser cache everywhere else (duration: 00m 18s) | [production] | 
            
  | 02:57 | <Krinkle> | sync-common failed on mw1010.eqiad.wmnet | [production] |