| 2012-08-15
      
      § | 
    
  | 05:22 | <Tim> | moving MediaWiki logs to fluorine | [production] | 
            
  | 05:17 | <tstarling> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 05:15 | <tstarling> | synchronized wmf-config/CommonSettings.php | [production] | 
            
  | 04:38 | <Tim> | added rate limit exemption for 27.108.200.111 | [production] | 
            
  | 04:38 | <tstarling> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 03:00 | <Tim> | built a udplog package for precise and uploaded it | [production] | 
            
  | 02:35 | <tstarling> | synchronized wmf-config/CommonSettings.php  'enabling Scribunto and CodeEditor on test2wiki' | [production] | 
            
  | 02:28 | <tstarling> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 02:15 | <tstarling> | Finished syncing Wikimedia installation... : | [production] | 
            
  | 02:01 | <LocalisationUpdate> | failed: git pull of extensions failed | [production] | 
            
  | 01:46 | <tstarling> | Started syncing Wikimedia installation... : | [production] | 
            
  | 01:00 | <tstarling> | Started syncing Wikimedia installation... : | [production] | 
            
  | 00:47 | <tstarling> | Started syncing Wikimedia installation... : | [production] | 
            
  | 00:39 | <Tim> | running scap to for extension-list update | [production] | 
            
  
    | 2012-08-14
      
      § | 
    
  | 23:53 | <catrope> | synchronized wmf-config/CommonSettings.php  'Move VE Parsoid from cadmium to wtp1' | [production] | 
            
  | 21:25 | <kaldari> | synchronized php-1.20wmf9/extensions/PageTriage/tools/cleanupPageTriage.php  'adding PageTriage maintainence script' | [production] | 
            
  | 21:05 | <kaldari> | Finished syncing Wikimedia installation... : | [production] | 
            
  | 20:45 | <kaldari> | Started syncing Wikimedia installation... : | [production] | 
            
  | 20:38 | <mutante> | killed duplicate nagios-wm | [production] | 
            
  | 20:20 | <cmjohnson1> | db44 swapping out disk1 rt 3424 | [production] | 
            
  | 20:08 | <ottomata> | installed precise and puppetized stat1001 | [production] | 
            
  | 19:26 | <cmjohnson1> | search23 shutting down to replace DIMM A2 rt 3423 | [production] | 
            
  | 19:00 | <catrope> | synchronized wmf-config/InitialiseSettings.php  'Deploying https://gerrit.wikimedia.org/r/19399' | [production] | 
            
  | 18:52 | <RoanKattouw> | Running /root/fixownership < /root/badownershipfiles in a screen on ms7 | [production] | 
            
  | 18:14 | <notpeter> | stopping puppet on brewster for partman experimentation | [production] | 
            
  | 17:34 | <catrope> | synchronized php-1.20wmf9/.gitmodules | [production] | 
            
  | 17:33 | <RoanKattouw> | Yesterday's find on ms7 is done, found 1.8 million files with bad ownership. Will run a batch chown on them soon | [production] | 
            
  | 17:32 | <catrope> | synchronized php-1.20wmf9/extensions/UploadWizard  'Deploy 9d9189458ad8d966d952d295918847f00e304cb5' | [production] | 
            
  | 09:17 | <nikerabbit> | synchronized php-1.20wmf9/extensions/Translate/  'I18ndeploy: Translate bugs' | [production] | 
            
  | 07:46 | <tstarling> | Finished syncing Wikimedia installation... : | [production] | 
            
  | 07:07 | <tstarling> | Started syncing Wikimedia installation... : | [production] | 
            
  | 06:48 | <Tim> | running scap to push out CodeEditor and Scribunto sources, not enabled yet | [production] | 
            
  | 05:45 | <olivneh> | synchronized php-1.20wmf9/extensions/E3Experiments/  'Update E3Experiments to master' | [production] | 
            
  | 03:34 | <Tim> | restarting all apaches to get LuaSandbox extension | [production] | 
            
  | 03:26 | <Tim> | on mw8: removed broken package php5-memcached | [production] | 
            
  | 02:45 | <LocalisationUpdate> | completed (1.20wmf8) at Tue Aug 14 02:45:30 UTC 2012 | [production] | 
            
  | 02:28 | <LocalisationUpdate> | completed (1.20wmf9) at Tue Aug 14 02:28:13 UTC 2012 | [production] | 
            
  | 01:06 | <Tim> | restarting apaches to get new wmerrors configuration | [production] | 
            
  
    | 2012-08-13
      
      § | 
    
  | 23:53 | <Tim> | uploaded a new version of the php5-wmerrors package for lucid and precise | [production] | 
            
  | 23:18 | <mutante> | removed #wikimedia-tech from /usr/local/bin/start-nagios-bot | [production] | 
            
  | 23:01 | <mutante> | killing nagios-wm process which isn't supposed to be on #wikimedia-tech anymore | [production] | 
            
  | 21:58 | <cmjohnson1> | running authdns-update for labsdb1 & 2 | [production] | 
            
  | 21:19 | <RoanKattouw> | ...piping result into ms7:/root/badownershipfiles | [production] | 
            
  | 21:18 | <RoanKattouw> | ms7# find /export/upload/wik*/*/{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f,archive,math,temp,timeline} ! -user apache | [production] | 
            
  | 21:17 | <RoanKattouw> | Searching for files with wrong ownership on ms7 using | [production] | 
            
  | 21:13 | <RobH> | reinstalling virt1004 to see if the raid issue re-appears | [production] | 
            
  | 20:55 | <notpeter> | updating root auth keys file on ms7 by hand. | [production] | 
            
  | 20:36 | <catrope> | synchronized php-1.20wmf9/resources/jquery/jquery.tablesorter.js  'Deploy 1fafaef3aa157f49cbbf4b9cffb03544bcf72a08' | [production] | 
            
  | 20:04 | <preilly> | updating zero for custom banner colors | [production] | 
            
  | 20:03 | <preilly> | synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess  'update for landing page' | [production] |