| 
      
        2010-12-13
      
      §
     | 
  
    
  | 20:10 | 
  <Ryan_Lane> | 
  moving payments4 into squid vlan | 
  [production] | 
            
  | 20:09 | 
  <mark> | 
  Failed over text.esams and bits.esams traffic to amslvs3 using an increased BGP metric for amslvs1 routes on csw1-esams | 
  [production] | 
            
  | 20:03 | 
  <mark> | 
  Configured amslvs3 with option conn_tab_size=20 for module ip_vs | 
  [production] | 
            
  | 20:01 | 
  <mark> | 
  Rebooting amslvs3 again to test | 
  [production] | 
            
  | 19:53 | 
  <mark> | 
  Rebooting amslvs3 | 
  [production] | 
            
  | 19:51 | 
  <mark> | 
  Running apt-get upgrade on amslvs3 | 
  [production] | 
            
  | 19:50 | 
  <mark> | 
  Added "kernel-ppa" PPA to amslvs3, installing linux 2.6.36 | 
  [production] | 
            
  | 18:34 | 
  <Ryan_Lane> | 
  renaming payments3 and payments4 in puppet | 
  [production] | 
            
  | 18:27 | 
  <Ryan_Lane> | 
  restarting network and services on payments3 | 
  [production] | 
            
  | 18:00 | 
  <Ryan_Lane> | 
  moving payments3 and payments4 to a public subnet | 
  [production] | 
            
  | 17:59 | 
  <Ryan_Lane> | 
  disabling payments3 and payments4 from haproxy config on loudon | 
  [production] | 
            
  | 12:42 | 
  <mark> | 
  Revived down servers | 
  [production] | 
            
  
    | 
      
        2010-12-12
      
      §
     | 
  
    
  | 22:01 | 
  <aaron> | 
  synchronized php-1.5/wmf-config/flaggedrevs.php  'removed dead settings' | 
  [production] | 
            
  | 18:47 | 
  <apergos> | 
  bayes unreachable by mgmt or ping, needs power cycled somehow | 
  [production] | 
            
  | 18:47 | 
  <apergos> | 
  bringing snapshot1 back up (not intended, but it was sitting there in the bios, I was actually looking for the bayes mgmt console as its current ip for that is nonresponsive... seems like snapshot1 got the old one though.  | 
  [production] | 
            
  | 15:55 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'Belated style version bump for last sync' | 
  [production] | 
            
  | 15:38 | 
  <catrope> | 
  synchronized wmf-deployment/extensions/UploadWizard/resources/combined.min.js  'r78249' | 
  [production] | 
            
  | 14:26 | 
  <aaron> | 
  synchronized php-1.5/wmf-config/flaggedrevs.php  'Reorganized autopromote settings to be more sane' | 
  [production] | 
            
  | 05:52 | 
  <Ryan_Lane> | 
  installing php5-uuid on nova-controller.tesla | 
  [production] | 
            
  
    | 
      
        2010-12-10
      
      §
     | 
  
    
  | 23:29 | 
  <Ryan_Lane> | 
  adding generic::geoip to owa* | 
  [production] | 
            
  | 20:24 | 
  <RobH> | 
  updating dns with emery entry | 
  [production] | 
            
  | 20:20 | 
  <RobH> | 
  snapshot1 powering back up, emery being pulled for temp storage use | 
  [production] | 
            
  | 18:54 | 
  <RobH> | 
  snapshot1 going offline to add capacity | 
  [production] | 
            
  | 13:40 | 
  <mark> | 
  Deployed new version exim 4.69-2ubuntu0.1wm2 with a str_format vuln fixed from exim 4.70 (lily, mchenry, sanger, grosley, williams) | 
  [production] | 
            
  | 13:40 | 
  <mark> | 
  Ran apt-get upgrade on sockpuppet | 
  [production] | 
            
  | 13:40 | 
  <mark> | 
  Ran apt-get upgrade on lily | 
  [production] | 
            
  | 05:57 | 
  <Ryan_Lane> | 
  cleaning /tmp on pdf1-3 | 
  [production] | 
            
  | 05:45 | 
  <Ryan_Lane> | 
  rebooting pdf3 | 
  [production] | 
            
  | 05:38 | 
  <Ryan_Lane> | 
  running mw-serve-ctl in cleanup mode on pdf3 | 
  [production] | 
            
  | 02:10 | 
  <Ryan_Lane> | 
  installing ldap scripts into /usr/local/sbin on nova-controller.tesla | 
  [production] | 
            
  | 02:03 | 
  <Ryan_Lane> | 
  installing ldap-utils on nova-controller.tesla | 
  [production] | 
            
  | 01:06 | 
  <Ryan_Lane> | 
  installing php5-curl on nova-controller.tesla | 
  [production] | 
            
  | 00:27 | 
  <Ryan_Lane> | 
  installing php5-memcache on nova-controller.tesla | 
  [production] | 
            
  | 00:27 | 
  <Ryan_Lane> | 
  installing php-apc and memcached on nova-controller.tesla | 
  [production] | 
            
  | 00:18 | 
  <mark> | 
  Deployed slightly modified exim4-daemon-heavy 4.69-2ubuntu0.1wm1 package on mchenry, sanger, lily, grosley | 
  [production] | 
            
  
    | 
      
        2010-12-09
      
      §
     | 
  
    
  | 21:50 | 
  <awjrichards> | 
  synchronized php-1.5/extensions/ContributionTracking/ContributionTracking.i18n.php  'Updating i18n file to allow for translatable item-name for paypal payments' | 
  [production] | 
            
  | 21:10 | 
  <awjrichards> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'Re-updating common settings to use new configuration style  and to play nicely with new contribution-tracking-setup.php' | 
  [production] | 
            
  | 21:02 | 
  <awjrichards> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'Fixing config revert due to misspelled extension path' | 
  [production] | 
            
  | 21:01 | 
  <awjrichards> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'Rolling back config changes do to config problems - looking for database in wrong place' | 
  [production] | 
            
  | 20:56 | 
  <awjrichards> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'Updating ContributionTracking configuration to use  style and also updating config vars for ContributionTracking to use new recurring payments IPN listener' | 
  [production] | 
            
  | 20:55 | 
  <awjrichards> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'Updating ContributionTracking configuration to use  style and also updating config vars for ContributionTracking to use new recurring payments IPN listener' | 
  [production] | 
            
  | 20:48 | 
  <RobH> | 
  updating dns for storage3 | 
  [production] | 
            
  | 20:47 | 
  <aaron> | 
  synchronized php-1.5/wmf-config/flaggedrevs.php  'removed unused config var' | 
  [production] | 
            
  | 20:37 | 
  <richcole> | 
  rebooting dataset1 to run scripts | 
  [production] | 
            
  | 20:36 | 
  <awjrichards> | 
  synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php  'Updating ContributionTracking extension to be able to handle recurring donations via PayPal' | 
  [production] | 
            
  | 20:36 | 
  <awjrichards> | 
  synchronized php-1.5/extensions/ContributionTracking/ContributionTracking.php  'Updating ContributionTracking extension to be able to handle recurring donations via PayPal' | 
  [production] | 
            
  | 20:36 | 
  <awjrichards> | 
  synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_OWA_ref.sql  'Updating ContributionTracking extension to be able to handle recurring donations via PayPal' | 
  [production] | 
            
  | 20:35 | 
  <awjrichards> | 
  synchronized php-1.5/extensions/ContributionTracking/patch-owa.sql  'Updating ContributionTracking extension to be able to handle recurring donations via PayPal' | 
  [production] | 
            
  | 20:35 | 
  <awjrichards> | 
  synchronized php-1.5/extensions/ContributionTracking/ContributionTracking.sql  'Updating ContributionTracking extension to be able to handle recurring donations via PayPal' | 
  [production] | 
            
  | 20:35 | 
  <awjrichards> | 
  synchronized php-1.5/extensions/ContributionTracking/ContributionTracking.pg.sql  'Updating ContributionTracking extension to be able to handle recurring donations via PayPal' | 
  [production] |