| 
      
        2010-12-25
      
      §
     | 
  
    
  | 23:39 | 
  <apergos> | 
  ran Platonides' script "purgeStaleMemcachedText.php" on s3/s7 projects, see /home/wikipedia/common/wmf-deployment/maintenance/purgeStaleMemcachedText.php on fenari (not added to the local branch yet)  | 
  [production] | 
            
  | 21:14 | 
  <ariel> | 
  synchronized wmf-deployment/wmf-config/db.php  'read-only for s3/s7 while we sort out edit sync and cache issues' | 
  [production] | 
            
  | 20:17 | 
  <mark> | 
  New master bin-log db17-bin.003, position 406 | 
  [production] | 
            
  | 20:16 | 
  <mark> | 
  Switched master on s3, s7 from db27 to db17, just one slave db25 | 
  [production] | 
            
  | 20:16 | 
  <mark> | 
  synchronized php-1.5/wmf-config/db.php  'Going read-write on s3, s7' | 
  [production] | 
            
  | 20:13 | 
  <mark> | 
  synchronized php-1.5/wmf-config/db.php  'Switch to new master db17, slave db25 for s3/s7' | 
  [production] | 
            
  | 20:03 | 
  <mark> | 
  synchronized php-1.5/wmf-config/db.php  | 
  [production] | 
            
  | 20:02 | 
  <mark> | 
  synchronized php-1.5/wmf-config/db.php  'Remove db39 from rotation' | 
  [production] | 
            
  | 19:58 | 
  <mark> | 
  Switching master to db17 | 
  [production] | 
            
  | 19:58 | 
  <mark> | 
  synchronized php-1.5/wmf-config/db.php  | 
  [production] | 
            
  | 19:43 | 
  <Ryan_Lane> | 
  purging old bin logs on db27 | 
  [production] | 
            
  | 14:32 | 
  <apergos> | 
  turned off Indexes option on wikitech in apache config files, just like on the cluster | 
  [production] | 
            
  | 12:18 | 
  <apergos> | 
  powercycling searchidx1, can't get in through management console, nonresponsive to ssh, google says the weird ilom messages are a firmware issue, so... reset. | 
  [production] | 
            
  | 11:11 | 
  <rainman-sr> | 
  searchidx1 is down for about 5 days now, can someone take a look at it? Cannot ssh to it. Also search group dissapeared from ganglia. | 
  [production] | 
            
  | 00:22 | 
  <mark> | 
  Running apt-get dist-upgrade on mobile3 | 
  [production] | 
            
  | 00:21 | 
  <mark> | 
  Rebalanced LVS weights of mobile servers on lvs4 from 20/50/30 to 30/50/50 | 
  [production] | 
            
  | 00:06 | 
  <mark> | 
  powercycled mobile3 | 
  [production] | 
            
  
    | 
      
        2010-12-23
      
      §
     | 
  
    
  | 18:23 | 
  <apergos> | 
  added export of /data for download host to class in puppet | 
  [production] | 
            
  | 17:57 | 
  <Ryan_Lane> | 
  ignore that last message | 
  [production] | 
            
  | 17:57 | 
  <Cobi> | 
  TODO: Give Cobi root on servers | 
  [production] | 
            
  | 17:54 | 
  <Ryan_Lane> | 
  added vivek and aditya to nova-controller.tesla, nova-compute1, nova-compute2, and added them to sudoers | 
  [production] | 
            
  | 17:46 | 
  <apergos> | 
  added misc::download class for dataset2 in puppet (handles lighttpd for download.wm)  | 
  [production] | 
            
  | 14:02 | 
  <hcatlin> | 
  deploying code update to m. | 
  [production] | 
            
  | 10:42 | 
  <tfinc> | 
  synchronized php-1.5/wmf-config/reporting-setup.php  'Flipping back to db10 since weve cleared up our space issues' | 
  [production] | 
            
  | 01:39 | 
  <apergos> | 
  dataset2 racked and installed (thanks rich, robh, mark).  now copying data back off tridge, rsync in screen session from tridge as root. | 
  [production] | 
            
  | 00:40 | 
  <tfinc> | 
  synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php  | 
  [production] | 
            
  | 00:40 | 
  <tfinc> | 
  synchronized php-1.5/extensions/ContributionTracking/ContributionTracking.i18n.php  | 
  [production] | 
            
  | 00:23 | 
  <tomaszf_> | 
  updating IPN listener on grosley to r83 | 
  [production] | 
            
  
    | 
      
        2010-12-22
      
      §
     | 
  
    
  | 23:58 | 
  <tomaszf> | 
  updating civicrm thank_you module to pick up email fix | 
  [production] | 
            
  | 23:35 | 
  <nelson__> | 
  snow | 
  [production] | 
            
  | 23:28 | 
  <tomaszf> | 
  updating thank_you and queue2civicrm module on grosley | 
  [production] | 
            
  | 23:28 | 
  <Ryan_Lane> | 
  started profile-collector on spence | 
  [production] | 
            
  | 23:21 | 
  <tfinc> | 
  synchronized php-1.5/extensions/LandingCheck/SpecialLandingCheck.php  | 
  [production] | 
            
  | 22:05 | 
  <demon> | 
  synchronizing Wikimedia installation... Revision: 78853: Deploy various FR/PC changes, r78852 | 
  [production] | 
            
  | 21:30 | 
  <RobH> | 
  udpated dns with dataset2 info | 
  [production] | 
            
  | 21:24 | 
  <tomaszf> | 
  fixing perms on grosley:/srv/org.wikimedia.civicrm/sites/all/modules to point to www-data instead of awjrichards | 
  [production] | 
            
  | 16:30 | 
  <Ryan_Lane> | 
  apache-graceful-all was me ;) | 
  [production] | 
            
  | 16:23 | 
  <Ryan_Lane> | 
  reloading apache on srv*, incrementally | 
  [production] | 
            
  | 15:48 | 
  <catrope> | 
  synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics_body.php  'r78799' | 
  [production] | 
            
  | 15:48 | 
  <catrope> | 
  synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.php  'r78799' | 
  [production] | 
            
  | 11:22 | 
  <Tim> | 
  added Roan to security@wikimedia.org alias list | 
  [production] |