| 
      
        2015-09-29
      
      §
     | 
  
    
  | 02:25 | 
  <bblack> | 
  ipv6 flap experiment: raise ipv6/route/max_size from 4096 to 131072 manually on cp*, actually | 
  [production] | 
            
  | 02:24 | 
  <bblack> | 
  ipv6 flap experiment: raise ipv6/route/max_size from 4096 to 131072 manually on cp20* | 
  [production] | 
            
  | 01:55 | 
  <robh> | 
  upload-lb.codfw.wikimedia.org_ipv6 page flap | 
  [production] | 
            
  | 01:43 | 
  <catrope@tin> | 
  Synchronized wmf-config/CommonSettings.php: Plumbing for wmgVisualEditorTransitionDefault (duration: 00m 17s) | 
  [production] | 
            
  | 01:42 | 
  <catrope@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Add wmgVisualEditorTransitionDefault (false everywhere) (duration: 00m 17s) | 
  [production] | 
            
  | 01:13 | 
  <robh> | 
  didnt powercycle analytics1035 yet, it recovered on its own. | 
  [production] | 
            
  | 01:12 | 
  <robh> | 
  powercycling analytics1035, seems oom, cannot login via ssh or serial console | 
  [production] | 
            
  | 00:56 | 
  <tstarling@tin> | 
  Synchronized php-1.26wmf24/extensions/ParsoidBatchAPI: Fix fatal error I77fd7e8 (duration: 00m 17s) | 
  [production] | 
            
  
    | 
      
        2015-09-28
      
      §
     | 
  
    
  | 23:47 | 
  <ebernhardson@tin> | 
  Synchronized wmf-config/CommonSettings.php: Update ttmserver configuration to match elasticsearch security profile (duration: 00m 17s) | 
  [production] | 
            
  | 23:43 | 
  <ori@tin> | 
  Synchronized wmf-config/CommonSettings.php: I56db35b: Removed ignore_user_abort( true ) line (duration: 00m 18s) | 
  [production] | 
            
  | 23:13 | 
  <catrope@tin> | 
  Synchronized php-1.26wmf24/extensions/Echo: SWAT (duration: 00m 18s) | 
  [production] | 
            
  | 23:13 | 
  <catrope@tin> | 
  Synchronized php-1.26wmf24/extensions/Flow: SWAT (duration: 00m 19s) | 
  [production] | 
            
  | 23:12 | 
  <catrope@tin> | 
  Synchronized php-1.26wmf24/extensions/CentralNotice: SWAT (duration: 00m 19s) | 
  [production] | 
            
  | 20:51 | 
  <bearND> | 
  MobileApps deployed sha1  9df72ec | 
  [production] | 
            
  | 20:42 | 
  <subbu> | 
  deployed parsoid version b9e5244e + hotfix on tin to turn off batching api use since canary restart of wtp1002 showed some batching api errors | 
  [production] | 
            
  | 20:26 | 
  <yuvipanda> | 
  depooled cp2017 from pybal config too | 
  [production] | 
            
  | 20:22 | 
  <yuvipanda> | 
  depooled cp2017 since it's down | 
  [production] | 
            
  | 19:02 | 
  <chasemp> | 
  powercycle iridium via console as it's unresponsive | 
  [production] | 
            
  | 15:50 | 
  <krenair@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241579/ (duration: 00m 17s) | 
  [production] | 
            
  | 15:28 | 
  <krenair@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241598/ (duration: 00m 17s) | 
  [production] | 
            
  | 15:11 | 
  <krenair@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241649/ (duration: 00m 17s) | 
  [production] | 
            
  | 14:00 | 
  <paravoid> | 
  running failover eqiad->codfw test for all frontend traffic | 
  [production] | 
            
  | 06:11 | 
  <ori@tin> | 
  Synchronized wmf-config/CommonSettings.php: I179be4bd3: Rely on timeouts specified in php.ini rather than calling ini_set() (duration: 00m 17s) | 
  [production] | 
            
  | 05:52 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Mon Sep 28 05:52:34 UTC 2015 (duration 52m 33s) | 
  [production] | 
            
  | 02:27 | 
  <l10nupdate@tin> | 
  LocalisationUpdate completed (1.26wmf24) at 2015-09-28 02:27:16+00:00 | 
  [production] | 
            
  | 02:20 | 
  <l10nupdate@tin> | 
  Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 06m 54s) | 
  [production] | 
            
  
    | 
      
        2015-09-25
      
      §
     | 
  
    
  | 23:54 | 
  <ori@tin> | 
  Synchronized php-1.26wmf24/includes/GlobalFunctions.php: 37c6972f94: Made wfIsBadImage() use APC (duration: 00m 17s) | 
  [production] | 
            
  | 22:16 | 
  <mutante> | 
  sodium - shutting down | 
  [production] | 
            
  | 21:53 | 
  <ori> | 
  Armed new servicedeploy_rsa on mira | 
  [production] | 
            
  | 21:51 | 
  <ori> | 
  Armed new servicedeploy_rsa on tin | 
  [production] | 
            
  | 20:45 | 
  <urandom> | 
  starting a Cassandra repair on xeon (nodetool repair -pr) | 
  [production] | 
            
  | 19:30 | 
  <mutante> | 
  restarted apache on gallium | 
  [production] | 
            
  | 19:07 | 
  <bblack> | 
  configuring and enabling lvs-cross-row ports on asw-b-eqiad for lvs1007,8,10,11 | 
  [production] | 
            
  | 18:04 | 
  <jynus> | 
  performing schema change on m5-master "nova" | 
  [production] | 
            
  | 17:26 | 
  <jynus> | 
  restarting and upgrading db1051 mysql (depooled) | 
  [production] | 
            
  | 17:20 | 
  <godog> | 
  reboot ms-be1005, xfs | 
  [production] | 
            
  | 16:17 | 
  <akosiaris> | 
  duplicating database otrs to otrsupgradetest for testing the upgrade procedure | 
  [production] | 
            
  | 16:16 | 
  <cmjohnson1> | 
  cp1046 stopped icinga checks for hardware troubleshooting  | 
  [production] | 
            
  | 14:43 | 
  <moritzm> | 
  installed rpcbind and apport security updates on various servers | 
  [production] | 
            
  | 14:36 | 
  <krenair@tin> | 
  Synchronized php-1.26wmf24/includes/specials/SpecialMovepage.php: https://gerrit.wikimedia.org/r/#/c/241045/ (duration: 00m 17s) | 
  [production] |