| 2009-07-15
      
      § | 
    
  | 20:18 | <brion> | updated http://en.wikipedia.org/wiki/MediaWiki:Uploaddisabledtext & http://commons.wikimedia.org/wiki/MediaWiki:Uploaddisabledtext | [production] | 
            
  | 19:43 | <fvassard> | synchronized php-1.5/CommonSettings.php  'Disabling Uploads while ms1 gets fixed (again with an s after upload).' | [production] | 
            
  | 19:40 | <fvassard> | synchronized php-1.5/CommonSettings.php  'Disabling Uploads while ms1 gets fixed.' | [production] | 
            
  | 19:40 | <atglenn> | bringing solaris up to current patch level on ms1 | [production] | 
            
  | 19:34 | <brion> | Ok, we're going to temporarily shut off uploading and unmount the uploads dir while we muck about with ms1. | [production] | 
            
  | 19:14 | <brion> | dropping export/upload@daily-2009-07-11_03:10:00 | [production] | 
            
  | 19:08 | <brion> | restarting web server on ms1, see if that resets some connections to the backend scalers | [production] | 
            
  | 19:05 | <brion> | restarting nfsd on ms1 | [production] | 
            
  | 18:58 | <brion> | dropping zfs snapshot export/upload@daily-2009-07-09_03:10:00 | [production] | 
            
  | 18:25 | <RobH_A90> | drac and physical setup done for dump1,2,3, will install remotely | [production] | 
            
  | 17:52 | <RobH_A90> | updated dns for new dump processing servers public and management ips | [production] | 
            
  | 17:41 | <Fred> | bounced apache on srv45 | [production] | 
            
  | 17:37 | <Fred> | bounced apache on srv47 | [production] | 
            
  | 17:09 | <RobH_A90> | pdf1 is not coming back, working on it | [production] | 
            
  | 16:56 | <RobH_A90> | shutting down pdf1 and mobile1 to move their power too, weee | [production] | 
            
  | 16:55 | <RobH_A90> | shutting down spence to move | [production] | 
            
  | 16:50 | <RobH_A90> | shutting down singer to move its power, blogs and other associated services will be offline for approx. 5 minutes | [production] | 
            
  | 16:47 | <Andrew> | Restarting apache on prototype | [production] | 
            
  | 16:46 | <RobH_A90> | shutting down grosley for power move | [production] | 
            
  | 16:45 | <RobH_A90> | all these power moves are to add the new dump processing servers to the rack | [production] | 
            
  | 16:45 | <RobH_A90> | shutting down fenari for power move | [production] | 
            
  | 16:43 | <RobH_A90> | shut down eiximenis and erzurumi to move their power | [production] | 
            
  | 16:34 | <RobH_A90> | shutting down some servers and moving power around in a4-sdtpa | [production] | 
            
  | 16:19 | <Andrew> | Test | [production] | 
            
  | 16:17 | <Andrew> | Changed morebots to tell you through a channel message instead of a private notice when the logging is successful. | [production] | 
            
  | 15:54 | <Fred> | kernel updated on wikitech from 2.6.18.8 to 2.6.29 (latest available on linode) | [production] | 
            
  | 15:49 | <Andrew> | Fixed auto-submission of honeypot data, was broken because it needed my perl include path. | [production] | 
            
  | 15:40 | <Fred> | rebooting wikitech to install new kernel | [production] | 
            
  
    | 2009-07-14
      
      § | 
    
  | 23:15 | <brion> | synchronized php-1.5/InitialiseSettings.php  'fix fix to enwiki confirmed gruop :D' | [production] | 
            
  | 22:24 | <brion> | synchronized php-1.5/InitialiseSettings.php  'fix to confirmed group for en' | [production] | 
            
  | 20:44 | <brion> | synchronized wmf-deployment/cache/trusted-xff.cdb | [production] | 
            
  | 20:41 | <brion> | synchronized wmf-deployment/cache/trusted-xff.cdb | [production] | 
            
  | 20:40 | <brion> | synchronized wmf-deployment/AdminSettings.php | [production] | 
            
  | 20:22 | <Fred> | restarted a bunch of dead apaches | [production] | 
            
  | 20:10 | <brion> | doing a sync-common-all w/ attempt to put test.wikipedia on wmf-deployment branch | [production] | 
            
  | 19:50 | <robh> | synchronized php-1.5/InitialiseSettings.php | [production] | 
            
  | 19:11 | <robh> | synchronized php-1.5/InitialiseSettings.php  'bug 19611 forgot one thing' | [production] | 
            
  | 19:09 | <robh> | synchronized php-1.5/InitialiseSettings.php  'bug 19611' | [production] | 
            
  | 19:08 | <robh> | synchronized php-1.5/InitialiseSettings.php  'bug 19611' | [production] | 
            
  | 14:16 | <domas> | dropped all june snapshots on ms1, thus providing some relief | [production] | 
            
  | 01:52 | <river> | patched ms4 in preperation for upload copy | [production] | 
            
  
    | 2009-07-13
      
      § | 
    
  | 21:31 | <Rob> | pushing dns update to fix management ips for new apaches | [production] | 
            
  | 19:05 | <Fred> | added storage3 to ganglia monitor. | [production] | 
            
  | 18:50 | <brion> | synchronized php-1.5/abusefilter.php  'Disable dewiki missingsummary, mysteriously in abusefilter section. Per bug 19208' | [production] | 
            
  | 16:30 | <Fred> | install wikimedia-nis-client to srv66 and mounted /home. | [production] | 
            
  | 16:28 | <brion> | synchronized php-1.5/InitialiseSettings.php  'fixing wikispecies RC-IRC prefix to species.wikimedia' | [production] | 
            
  | 16:27 | <brion> | test wiki was apparently moved from dead srv35 to srv66, which has new NFS-less config. thus fail since test runs from nfs | [production] | 
            
  | 16:24 | <brion> | test wiki borked; reported down for several days now :) investigating | [production] | 
            
  | 15:12 | <midom> | synchronized php-1.5/db.php  'db26 raid issues' | [production] | 
            
  | 14:55 | <midom> | synchronized php-1.5/db.php  'db3 and db5 coming live as commons servers' | [production] |