| 2011-08-04
      
      § | 
    
  | 23:57 | <binasher> | deploying mobile squid acls with nokia and netfront enabled for a brief test to see if todays mobile gateway changes will allow it to support the load | [production] | 
            
  | 23:10 | <binasher> | squid mobile redirect deployment completed | [production] | 
            
  | 22:38 | <binasher> | mobile redirect deploy complete to all pmtpa squids. mobile cluster looks good, starting amsterdam | [production] | 
            
  | 22:17 | <binasher> | starting an extra slow rollout of mobile redirect acls to frontend squids, minus nokia and netfront | [production] | 
            
  | 21:43 | <binasher> | varnish 3.0 is now on all RoR mobile gw servers - only caching static non-article content | [production] | 
            
  | 21:33 | <binasher> | mobile2 is back / varnish for static content / apache relax | [production] | 
            
  | 19:52 | <binasher> | pulling mobile2 from lvs for testing | [production] | 
            
  | 17:17 | <reedy> | synchronized multiversion/MWMultiVersion.php  '[[rev:93902|r93902]]' | [production] | 
            
  | 17:17 | <reedy> | synchronized multiversion/MWMultiVersion.php  '[[rev:93901|r93901]]' | [production] | 
            
  | 17:12 | <reedy> | synchronized multiversion/MWMultiVersion.php  '[[rev:93901|r93901]]' | [production] | 
            
  | 17:11 | <AaronSchulz> | Fixed thumbnails on testwiki ([[bugzilla:30210|bug 30210]]) | [production] | 
            
  | 17:10 | <aaron> | synchronized live-1.5/MWVersion.php | [production] | 
            
  | 15:40 | <mark> | Mounted nas1001-a:/vol/images on ms1002:/vol/images for testing, started rsync copy from ms1002:/export/upload to /vol/images | [production] | 
            
  | 15:20 | <mark> | Created 12-disk aggregate test0 on nas1001-a | [production] | 
            
  | 15:01 | <RoanKattouw> | Staging some HTTPS support / prot rel URL code on testwiki, see today's 1.17wmf1 merges. DO NOT DEPLOY SITEWIDE until tomorrow | [production] | 
            
  | 13:19 | <RobH> | updated wikibugs on mchenry about 10 minutes ago and forgot to admin log | [production] | 
            
  | 13:18 | <mark> | Created new 64 bit aggregate and root volume on nas1001-a, copied over root data, destroyed old root volume, moving new aggregate to first disks | [production] | 
            
  | 13:00 | <RobH> | disabled line numbers in vimrc.local on puppet because i hate line numbers (it already says the line number yer on in the corner) | [production] | 
            
  | 12:21 | <mark> | Started disk replace operation of 0a.10.0 by 0a.10.3 and 0a.10.1 by 0a.10.4 on nas1-b | [production] | 
            
  | 12:12 | <mark> | Started disk replace operation of 0b.00.0 by 0a.01.1, and 0b.00.1 by 0a.01.2 on nas1-a | [production] | 
            
  | 12:08 | <mutante> | ragweed - package updates, restart required | [production] | 
            
  | 11:05 | <mutante> | amssq46 - power cycle, clean squid, dist-upgrade/kernel, reboot | [production] | 
            
  | 02:18 | <LocalisationUpdate> | completed at Thu Aug  4 02:20:48 UTC 2011 | [production] | 
            
  
    | 2011-08-03
      
      § | 
    
  | 21:22 | <binasher> | previous log line should have read $ppsize + 50 | [production] | 
            
  | 21:21 | <binasher> | dropping apache maxclients on mobile servers to ($ppsize - 50) | [production] | 
            
  | 18:24 | <aaron> | synchronized multiversion/MWMultiVersion.php  'deployed [[rev:93845|r93845]]' | [production] | 
            
  | 18:23 | <aaron> | synchronized multiversion/MWScript.php  'deployed [[rev:93845|r93845]]' | [production] | 
            
  | 16:19 | <tfinc> | synchronized wmf-config/InitialiseSettings.php  'testing collectinos on hewiki' | [production] | 
            
  | 16:13 | <mutante> | installed Net::Stomp Perl module on spence via cpan shell which fixed RT #703 | [production] | 
            
  | 14:32 | <RobH> | updating dns for magnesium and zinc eqiad misc servers to public IPs RT#1245 | [production] | 
            
  | 13:24 | <tstarling> | synchronized php-1.17/extensions/CodeReview/ui/WordCloud.php | [production] | 
            
  | 13:24 | <tstarling> | synchronized php-1.17/extensions/CodeReview/CodeReview.php | [production] | 
            
  | 12:21 | <mark> | Rebooting both nas1001 controllers with corrected network config | [production] | 
            
  | 10:02 | <root> | synchronized wmf-config/mc.php  'swapped out all down servers returning 0 with ones returning 100, cleaned up spares and down list to be accurate' | [production] | 
            
  | 09:44 | <RobH> | swapping out down memcached servers iwth tested up ones | [production] | 
            
  | 09:44 | <RobH> | incorrect, turns out there are some down, they reply but do not return full weight | [production] | 
            
  | 09:44 | <robh> | synchronized wmf-config/mc.php  'replacing down servers' | [production] | 
            
  | 09:40 | <RobH> | nagios shows spence memcached error, yet manual testing shows all memcached are pooled fine. | [production] | 
            
  | 09:01 | <Ryan_Lane> | installing ssl1001-4 | [production] | 
            
  | 08:09 | <apergos> | live hack to /usr/share/etherpad/etherpad/bin/run-local.sh on hooper to force maxthreads to 512 for etherpad.  restarted etherpad, this may or may not help connection issues (maybe we'll just oom instead?) | [production] | 
            
  | 07:40 | <domas> | stupid bot and its quotes, http://p.defau.lt/?UUberPb2hRtPxOzG7Q6OwQ is the url | [production] | 
            
  | 07:39 | <midom> | synchronized php/includes/db/DatabaseMysql.php  'livehacking out sql_mode and set charset code - http://p.defau.lt/?UUberPb2hRtPxOzG7Q6OwQ' | [production] | 
            
  | 02:20 | <LocalisationUpdate> | completed at Wed Aug  3 02:22:48 UTC 2011 | [production] |