| 
      
        2015-04-25
      
      §
     | 
  
    
  | 15:26 | 
  <subbu> | 
  deployed parsoid version fca17070 (cherry-pick of d2135c6b on parsoid master) | 
  [production] | 
            
  | 09:57 | 
  <_joe_> | 
  nuked User:Niteshift/MVneu/2015_April_21-30 on commonswiki | 
  [production] | 
            
  | 05:18 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Sat Apr 25 05:17:41 UTC 2015 (duration 17m 40s) | 
  [production] | 
            
  | 04:30 | 
  <mattflaschen> | 
  Synchronized wmf-config/CommonSettings-labs.php: Sync Beta Cluster-only change (for MW UI beta feature) (duration: 00m 16s) | 
  [production] | 
            
  | 04:30 | 
  <mattflaschen> | 
  Synchronized wmf-config/InitialiseSettings-labs.php: Sync Beta Cluster-only change (for MW UI beta feature) (duration: 00m 16s) | 
  [production] | 
            
  | 02:43 | 
  <LocalisationUpdate> | 
  completed (1.26wmf3) at 2015-04-25 02:41:54+00:00 | 
  [production] | 
            
  | 02:39 | 
  <l10nupdate> | 
  Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 05m 56s) | 
  [production] | 
            
  | 02:24 | 
  <LocalisationUpdate> | 
  completed (1.26wmf2) at 2015-04-25 02:23:33+00:00 | 
  [production] | 
            
  | 02:20 | 
  <l10nupdate> | 
  Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 07m 48s) | 
  [production] | 
            
  
    | 
      
        2015-04-24
      
      §
     | 
  
    
  | 22:14 | 
  <krinkle> | 
  Synchronized php-1.26wmf2/includes/resourceloader/ResourceLoaderModule.php: Ibedc31659ed (duration: 00m 14s) | 
  [production] | 
            
  | 22:13 | 
  <krinkle> | 
  Synchronized php-1.26wmf3/includes/resourceloader/ResourceLoaderModule.php: Ibedc31659ed (duration: 00m 17s) | 
  [production] | 
            
  | 21:11 | 
  <ottomata> | 
  started hdfs balancer run | 
  [production] | 
            
  | 20:34 | 
  <ori> | 
  Deployed I1fa012ca1: HHVM: Limit wall execution time of FCGI reqs to 290s | 
  [production] | 
            
  | 19:53 | 
  <aaron> | 
  Synchronized wmf-config/db-codfw.php: Removed unused "max threads" stuff (duration: 00m 15s) | 
  [production] | 
            
  | 19:52 | 
  <subbu> | 
  revert parsoid deploy to 3311936a | 
  [production] | 
            
  | 19:52 | 
  <aaron> | 
  Synchronized wmf-config/db-eqiad.php: Removed unused "max threads" stuff (duration: 00m 14s) | 
  [production] | 
            
  | 19:42 | 
  <demon> | 
  Synchronized php-1.26wmf2/extensions/CirrusSearch/includes/Searcher.php: undo debugging (duration: 00m 14s) | 
  [production] | 
            
  | 19:40 | 
  <demon> | 
  Synchronized php-1.26wmf2/extensions/CirrusSearch/includes/Searcher.php: debugging (duration: 00m 17s) | 
  [production] | 
            
  | 18:58 | 
  <ori> | 
  restarted puppetmaster on palladium as well | 
  [production] | 
            
  | 18:56 | 
  <ori> | 
  restarted apache2 on palladium | 
  [production] | 
            
  | 16:06 | 
  <andrewbogott> | 
  dist-upgrade (including kernel upgrade to 3.13.0-49-generic) on labvirt1004, rebooting | 
  [production] | 
            
  | 15:56 | 
  <demon> | 
  Synchronized wmf-config/: logging cleanup, mostly for labs (duration: 00m 21s) | 
  [production] | 
            
  | 15:42 | 
  <andrewbogott> | 
  dist-upgrade (including kernel upgrade to 3.13.0-49-generic) on labvirt1003, rebooting | 
  [production] | 
            
  | 15:08 | 
  <andrewbogott> | 
  dist-upgrade (including kernel upgrade to 3.13.0-49-generic) on labvirt1005, rebooting | 
  [production] | 
            
  | 14:24 | 
  <andrewbogott> | 
  dist-upgrade (including kernel upgrade to 3.13.0-49-generic) on labvirt1006, rebooting | 
  [production] | 
            
  | 10:31 | 
  <akosiaris> | 
  nova migrated a couple of etcd's project VMs | 
  [production] | 
            
  | 09:09 | 
  <_joe_> | 
  parsoid restart done | 
  [production] | 
            
  | 08:59 | 
  <_joe_> | 
  restarting parsoid cluster-wide | 
  [production] | 
            
  | 08:47 | 
  <ori> | 
  deployed parsoid/deploy 8b5de6aba / I4d55f6d50: Bump src to d2135c6b69 for deploy | 
  [production] | 
            
  | 08:09 | 
  <tstarling> | 
  Synchronized php-1.26wmf2/includes/filerepo/file/LocalFile.php: reverting live hack (duration: 00m 16s) | 
  [production] | 
            
  | 06:40 | 
  <ori> | 
  nuked http://commons.wikimedia.org/wiki/User:Niteshift/MVneu/2015_April_21-30 | 
  [production] | 
            
  | 05:44 | 
  <ori> | 
  Synchronized php-1.26wmf1/includes/filerepo/file/LocalFile.php: Undo local hack on version that is inactive (1.26wmf1). No-op. (duration: 00m 17s) | 
  [production] | 
            
  | 05:35 | 
  <ori> | 
  restart hhvm on mw1222; locked up in pthread_cond_wait, backtrace: https://phabricator.wikimedia.org/P552 | 
  [production] | 
            
  | 05:28 | 
  <ori> | 
  nuked https://commons.wikimedia.org/wiki/User:Niteshift/MVneu/2015_April_21-20 | 
  [production] | 
            
  | 05:18 | 
  <ori> | 
  Synchronized wmf-config/InitialiseSettings.php: $wgExportAllowHistory default false, $wgExportMaxHistory default 1000 -> 10 (duration: 00m 16s) | 
  [production] | 
            
  | 05:05 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Fri Apr 24 05:04:42 UTC 2015 (duration 4m 41s) | 
  [production] | 
            
  | 04:47 | 
  <ori> | 
  Synchronized php-1.26wmf2/includes/filerepo/file/LocalFile.php: Short-circuit LocalFile::loadExtraFromDB in attempt to mitigate outage (duration: 00m 12s) | 
  [production] | 
            
  | 04:42 | 
  <springle> | 
  killing LocalFile::loadExtraFromDB wholesale on s4 | 
  [production] | 
            
  | 04:32 | 
  <ori> | 
  Synchronized php-1.26wmf1/includes/filerepo/file/LocalFile.php: Short-circuit LocalFile::loadExtraFromDB in attempt to mitigate outage (duration: 00m 14s) | 
  [production] | 
            
  | 04:25 | 
  <ori> | 
  Did a cluster-wide 'service hhvm restart'. | 
  [production] | 
            
  | 02:48 | 
  <LocalisationUpdate> | 
  completed (1.26wmf3) at 2015-04-24 02:47:12+00:00 | 
  [production] | 
            
  | 02:44 | 
  <l10nupdate> | 
  Synchronized php-1.26wmf3/cache/l10n: (no message) (duration: 06m 00s) | 
  [production] | 
            
  | 02:30 | 
  <LocalisationUpdate> | 
  completed (1.26wmf2) at 2015-04-24 02:28:58+00:00 | 
  [production] | 
            
  | 02:25 | 
  <l10nupdate> | 
  Synchronized php-1.26wmf2/cache/l10n: (no message) (duration: 06m 35s) | 
  [production] | 
            
  | 00:47 | 
  <catrope> | 
  Synchronized wmf-config/CommonSettings.php: Revert RESTbase URL change (duration: 00m 13s) | 
  [production] | 
            
  | 00:21 | 
  <catrope> | 
  Synchronized php-1.26wmf3/extensions/VisualEditor: Fix RESTbase revid bug (duration: 00m 18s) | 
  [production] | 
            
  | 00:21 | 
  <catrope> | 
  Synchronized php-1.26wmf2/extensions/VisualEditor: Fix RESTbase revid bug (duration: 00m 17s) | 
  [production] | 
            
  | 00:07 | 
  <catrope> | 
  Synchronized wmf-config/CommonSettings.php: Use same-domain entry point for RESTbase (duration: 00m 13s) | 
  [production] |