| 
      
        2013-07-14
      
      §
     | 
  
    
  | 13:20 | 
  <mark> | 
  Upping PyBal weight to 15 for stolen appservers | 
  [production] | 
            
  | 13:19 | 
  <RoanKattouw> | 
  Hand-syncing CommonSettings.php with dsh to set $wgParsoidSkipRatio = 0 | 
  [production] | 
            
  | 13:17 | 
  <mark> | 
  Stealing 10 app servers, for the API pool | 
  [production] | 
            
  | 13:05 | 
  <catrope> | 
  synchronized wmf-config/CommonSettings.php  'Set $wgParsoidSkipRatio to 1 to let the API cluster breathe' | 
  [production] | 
            
  | 12:54 | 
  <RoanKattouw> | 
  Restarted Apache on mw1118, had to stop by hand using killall -9 (/etc/init.d/apache stop didn't work) | 
  [production] | 
            
  | 12:38 | 
  <RoanKattouw> | 
  Restarted pybal on lvs1003 | 
  [production] | 
            
  | 12:37 | 
  <RoanKattouw> | 
  Restarted pybal on lvs1006 (set api depool threshold to .4 from .6) | 
  [production] | 
            
  | 06:37 | 
  <apergos> | 
  powercycled mc1005, unreachable via console, from ganglia guess it was in swap death | 
  [production] | 
            
  | 02:07 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Sun Jul 14 02:07:10 UTC 2013 | 
  [production] | 
            
  | 02:02 | 
  <LocalisationUpdate> | 
  completed (1.22wmf10) at Sun Jul 14 02:02:31 UTC 2013 | 
  [production] | 
            
  | 02:01 | 
  <LocalisationUpdate> | 
  completed (1.22wmf9) at Sun Jul 14 02:01:43 UTC 2013 | 
  [production] | 
            
  
    | 
      
        2013-07-12
      
      §
     | 
  
    
  | 23:39 | 
  <gwicke> | 
  updated Parsoid to f6d3742 | 
  [production] | 
            
  | 23:37 | 
  <springle> | 
  restarted db32 slave | 
  [production] | 
            
  | 23:31 | 
  <springle> | 
  finished tag_summary rebuild | 
  [production] | 
            
  | 22:39 | 
  <aaron> | 
  cleared profiling data | 
  [production] | 
            
  | 22:33 | 
  <mutante> | 
  completely stopped and then started squid front and back on cp1001 (had socket already in use errors) | 
  [production] | 
            
  | 22:25 | 
  <springle> | 
  rebuilding tag_summary on enwiki | 
  [production] | 
            
  | 22:24 | 
  <mutante> | 
  restarting squid on cp1001 | 
  [production] | 
            
  | 22:06 | 
  <mutante> | 
  restarting job runners | 
  [production] | 
            
  | 21:06 | 
  <springle> | 
  stopped db32 mysql slave threads | 
  [production] | 
            
  | 19:56 | 
  <gwicke> | 
  synchronized php-1.22wmf9/extensions/Parsoid/php/ParsoidCacheUpdateJob.php  'updating Parsoid' | 
  [production] | 
            
  | 19:56 | 
  <gwicke> | 
  synchronized php-1.22wmf9/extensions/Parsoid/php/Parsoid.php  'updating Parsoid' | 
  [production] | 
            
  | 18:44 | 
  <RobH> | 
  authdns-update all ulsfo now 4XXX | 
  [production] | 
            
  | 18:19 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Remove wgOldChangeTagsIndex on all remaining wikis' | 
  [production] | 
            
  | 17:59 | 
  <gwicke> | 
  synchronized php-1.22wmf9/extensions/Parsoid/php/ParsoidCacheUpdateJob.php  'updating Parsoid master' | 
  [production] | 
            
  | 17:58 | 
  <gwicke> | 
  synchronized php-1.22wmf9/extensions/Parsoid/php/Parsoid.php  'updating Parsoid master' | 
  [production] | 
            
  | 16:26 | 
  <Krinkle> | 
  Graceful reload of Zuul to forward deployment to I4de1670e11d58e | 
  [production] | 
            
  | 16:09 | 
  <reedy> | 
  synchronized docroot and w | 
  [production] | 
            
  | 15:50 | 
  <reedy> | 
  synchronized docroot and w | 
  [production] | 
            
  | 15:34 | 
  <reedy> | 
  synchronized docroot and w | 
  [production] | 
            
  | 15:32 | 
  <reedy> | 
  synchronized docroot and w | 
  [production] | 
            
  | 15:30 | 
  <reedy> | 
  synchronized docroot and w | 
  [production] | 
            
  | 15:24 | 
  <reedy> | 
  synchronized docroot and w | 
  [production] | 
            
  | 15:19 | 
  <Krinkle> | 
  Resetting [[commons:User:CommonsDelinker]] account e-mailaddress (bug 51016) | 
  [production] | 
            
  | 10:42 | 
  <mark> | 
  Sending European wikidata and wikivoyage traffic to esams | 
  [production] | 
            
  | 05:12 | 
  <mutante> | 
  restarting ircecho on neon | 
  [production] | 
            
  | 03:02 | 
  <mwalker> | 
  and again... payments from 1c360e2cde650d1195f04aee294ddd3e0f5f9052 to 2a9169765b94e036c4d2ee0cada362e4de6e212e | 
  [production] | 
            
  | 02:47 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Fri Jul 12 02:47:23 UTC 2013 | 
  [production] | 
            
  | 02:40 | 
  <mwalker> | 
  attempting again on the payments cluster! from 1c360e2cde650d1195f04aee294ddd3e0f5f9052 to 7a5a3c376ed0b7f25c5fa0e9f14d6cc67e543982 | 
  [production] | 
            
  | 02:34 | 
  <LocalisationUpdate> | 
  completed (1.22wmf10) at Fri Jul 12 02:34:20 UTC 2013 | 
  [production] | 
            
  | 02:20 | 
  <mwalker> | 
  aaand nope; didn't work, reverting payments back to 1c360e2cde650d1195f04aee294ddd3e0f5f9052 | 
  [production] | 
            
  | 02:17 | 
  <mwalker> | 
  trying again -- updating payments from 1c360e2cde650d1195f04aee294ddd3e0f5f9052 to 8a801e809de37bc8595b19262faa9b9e47bfbd6e | 
  [production] | 
            
  | 02:16 | 
  <LocalisationUpdate> | 
  completed (1.22wmf9) at Fri Jul 12 02:16:00 UTC 2013 | 
  [production] |