| 
      
        2012-08-15
      
      §
     | 
  
    
  | 03:00 | 
  <Tim> | 
  built a udplog package for precise and uploaded it | 
  [production] | 
            
  | 02:35 | 
  <tstarling> | 
  synchronized wmf-config/CommonSettings.php  'enabling Scribunto and CodeEditor on test2wiki' | 
  [production] | 
            
  | 02:28 | 
  <tstarling> | 
  synchronized wmf-config/InitialiseSettings.php  | 
  [production] | 
            
  | 02:15 | 
  <tstarling> | 
  Finished syncing Wikimedia installation... :  | 
  [production] | 
            
  | 02:01 | 
  <LocalisationUpdate> | 
  failed: git pull of extensions failed | 
  [production] | 
            
  | 01:46 | 
  <tstarling> | 
  Started syncing Wikimedia installation... :  | 
  [production] | 
            
  | 01:00 | 
  <tstarling> | 
  Started syncing Wikimedia installation... :  | 
  [production] | 
            
  | 00:47 | 
  <tstarling> | 
  Started syncing Wikimedia installation... :  | 
  [production] | 
            
  | 00:39 | 
  <Tim> | 
  running scap to for extension-list update | 
  [production] | 
            
  
    | 
      
        2012-08-14
      
      §
     | 
  
    
  | 23:53 | 
  <catrope> | 
  synchronized wmf-config/CommonSettings.php  'Move VE Parsoid from cadmium to wtp1' | 
  [production] | 
            
  | 21:25 | 
  <kaldari> | 
  synchronized php-1.20wmf9/extensions/PageTriage/tools/cleanupPageTriage.php  'adding PageTriage maintainence script' | 
  [production] | 
            
  | 21:05 | 
  <kaldari> | 
  Finished syncing Wikimedia installation... :  | 
  [production] | 
            
  | 20:45 | 
  <kaldari> | 
  Started syncing Wikimedia installation... :  | 
  [production] | 
            
  | 20:38 | 
  <mutante> | 
  killed duplicate nagios-wm | 
  [production] | 
            
  | 20:20 | 
  <cmjohnson1> | 
  db44 swapping out disk1 rt 3424 | 
  [production] | 
            
  | 20:08 | 
  <ottomata> | 
  installed precise and puppetized stat1001 | 
  [production] | 
            
  | 19:26 | 
  <cmjohnson1> | 
  search23 shutting down to replace DIMM A2 rt 3423 | 
  [production] | 
            
  | 19:00 | 
  <catrope> | 
  synchronized wmf-config/InitialiseSettings.php  'Deploying https://gerrit.wikimedia.org/r/19399' | 
  [production] | 
            
  | 18:52 | 
  <RoanKattouw> | 
  Running /root/fixownership < /root/badownershipfiles in a screen on ms7 | 
  [production] | 
            
  | 18:14 | 
  <notpeter> | 
  stopping puppet on brewster for partman experimentation | 
  [production] | 
            
  | 17:34 | 
  <catrope> | 
  synchronized php-1.20wmf9/.gitmodules  | 
  [production] | 
            
  | 17:33 | 
  <RoanKattouw> | 
  Yesterday's find on ms7 is done, found 1.8 million files with bad ownership. Will run a batch chown on them soon | 
  [production] | 
            
  | 17:32 | 
  <catrope> | 
  synchronized php-1.20wmf9/extensions/UploadWizard  'Deploy 9d9189458ad8d966d952d295918847f00e304cb5' | 
  [production] | 
            
  | 09:17 | 
  <nikerabbit> | 
  synchronized php-1.20wmf9/extensions/Translate/  'I18ndeploy: Translate bugs' | 
  [production] | 
            
  | 07:46 | 
  <tstarling> | 
  Finished syncing Wikimedia installation... :  | 
  [production] | 
            
  | 07:07 | 
  <tstarling> | 
  Started syncing Wikimedia installation... :  | 
  [production] | 
            
  | 06:48 | 
  <Tim> | 
  running scap to push out CodeEditor and Scribunto sources, not enabled yet | 
  [production] | 
            
  | 05:45 | 
  <olivneh> | 
  synchronized php-1.20wmf9/extensions/E3Experiments/  'Update E3Experiments to master' | 
  [production] | 
            
  | 03:34 | 
  <Tim> | 
  restarting all apaches to get LuaSandbox extension | 
  [production] | 
            
  | 03:26 | 
  <Tim> | 
  on mw8: removed broken package php5-memcached | 
  [production] | 
            
  | 02:45 | 
  <LocalisationUpdate> | 
  completed (1.20wmf8) at Tue Aug 14 02:45:30 UTC 2012 | 
  [production] | 
            
  | 02:28 | 
  <LocalisationUpdate> | 
  completed (1.20wmf9) at Tue Aug 14 02:28:13 UTC 2012 | 
  [production] | 
            
  | 01:06 | 
  <Tim> | 
  restarting apaches to get new wmerrors configuration | 
  [production] | 
            
  
    | 
      
        2012-08-13
      
      §
     | 
  
    
  | 23:53 | 
  <Tim> | 
  uploaded a new version of the php5-wmerrors package for lucid and precise | 
  [production] | 
            
  | 23:18 | 
  <mutante> | 
  removed #wikimedia-tech from /usr/local/bin/start-nagios-bot  | 
  [production] | 
            
  | 23:01 | 
  <mutante> | 
  killing nagios-wm process which isn't supposed to be on #wikimedia-tech anymore | 
  [production] | 
            
  | 21:58 | 
  <cmjohnson1> | 
  running authdns-update for labsdb1 & 2 | 
  [production] | 
            
  | 21:19 | 
  <RoanKattouw> | 
  ...piping result into ms7:/root/badownershipfiles | 
  [production] | 
            
  | 21:18 | 
  <RoanKattouw> | 
  ms7# find /export/upload/wik*/*/{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f,archive,math,temp,timeline} ! -user apache | 
  [production] | 
            
  | 21:17 | 
  <RoanKattouw> | 
  Searching for files with wrong ownership on ms7 using | 
  [production] | 
            
  | 21:13 | 
  <RobH> | 
  reinstalling virt1004 to see if the raid issue re-appears | 
  [production] | 
            
  | 20:55 | 
  <notpeter> | 
  updating root auth keys file on ms7 by hand. | 
  [production] | 
            
  | 20:36 | 
  <catrope> | 
  synchronized php-1.20wmf9/resources/jquery/jquery.tablesorter.js  'Deploy 1fafaef3aa157f49cbbf4b9cffb03544bcf72a08' | 
  [production] | 
            
  | 20:04 | 
  <preilly> | 
  updating zero for custom banner colors | 
  [production] | 
            
  | 20:03 | 
  <preilly> | 
  synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess  'update for landing page' | 
  [production] | 
            
  | 20:03 | 
  <preilly> | 
  synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess  'update for landing page' | 
  [production] | 
            
  | 19:42 | 
  <preilly> | 
  synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess  'update for landing page' | 
  [production] | 
            
  | 19:42 | 
  <preilly> | 
  update to zero for banner text link | 
  [production] | 
            
  | 19:41 | 
  <preilly> | 
  synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess  'update for landing page' | 
  [production] | 
            
  | 19:36 | 
  <preilly> | 
  synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess  'update for landing page' | 
  [production] |