| 
      
        2012-08-27
      
      §
     | 
  
    
  | 23:31 | 
  <Ryan_Lane> | 
  restarting virt6 | 
  [production] | 
            
  | 23:08 | 
  <Ryan_Lane> | 
  upgrading virt6 to precise. This action will cause downtime for a while. | 
  [production] | 
            
  | 22:49 | 
  <mutante> | 
  fixed gmq.planet.wm updates | 
  [production] | 
            
  | 22:34 | 
  <Ryan_Lane> | 
  rebooting virt3 | 
  [production] | 
            
  | 22:24 | 
  <Ryan_Lane> | 
  upgrading virt3 to precise | 
  [production] | 
            
  | 22:23 | 
  <Ryan_Lane> | 
  disabling nova-compute on virt2 | 
  [production] | 
            
  | 21:01 | 
  <dschoon> | 
  ${whoami} | 
  [production] | 
            
  | 21:00 | 
  <Ryan_Lane> | 
  enabling OpenStackManager and LdapAuthentication on labsconsole temporarily for testing | 
  [production] | 
            
  | 21:00 | 
  <Ryan_Lane> | 
  enabling nova-compute on virt2 for testing instance creation | 
  [production] | 
            
  | 20:47 | 
  <ottomata> | 
  starting puppet back up on emery | 
  [production] | 
            
  | 20:32 | 
  <aaron> | 
  synchronized php-1.20wmf9/extensions/MobileFrontend  'deployed f2551fa97ba686101745d5cb97ef4678838cd3bb' | 
  [production] | 
            
  | 20:29 | 
  <aaron> | 
  synchronized php-1.20wmf10/extensions/MobileFrontend  'deployed 75104cf6d3bc6d528d2b85e09761005081fb5cf4' | 
  [production] | 
            
  | 20:21 | 
  <paravoid> | 
  deploying swift origs to all pmtpa backend squids | 
  [production] | 
            
  | 20:09 | 
  <Ryan_Lane> | 
  force running puppet on virt2 | 
  [production] | 
            
  | 20:04 | 
  <Ryan_Lane> | 
  rebooting virt2 | 
  [production] | 
            
  | 19:54 | 
  <paravoid> | 
  deploying swift origs on sq51 (1h ago), sq52, sq53 | 
  [production] | 
            
  | 19:54 | 
  <Ryan_Lane> | 
  upgrading virt2 to precise | 
  [production] | 
            
  | 19:38 | 
  <Ryan_Lane> | 
  upgrading OpenStackManager on labsconsole | 
  [production] | 
            
  | 19:33 | 
  <Ryan_Lane> | 
  adding keystone services and endpoints | 
  [production] | 
            
  | 19:32 | 
  <Ryan_Lane> | 
  upgrading nova, glance and keystone database schemas | 
  [production] | 
            
  | 19:18 | 
  <ottomata> | 
  temporarily stopping puppet on emery to test log2udp AFT clicktracking relay to vanadium | 
  [production] | 
            
  | 19:06 | 
  <aaron> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf10 | 
  [production] | 
            
  | 18:41 | 
  <Ryan_Lane> | 
  force running puppet on virt0 | 
  [production] | 
            
  | 18:36 | 
  <Ryan_Lane> | 
  restarting virt0 | 
  [production] | 
            
  | 18:09 | 
  <Ryan_Lane> | 
  upgrading virt0 to precise | 
  [production] | 
            
  | 17:54 | 
  <Ryan_Lane> | 
  running a fresh backup of opendj on virt0 | 
  [production] | 
            
  | 17:54 | 
  <Ryan_Lane> | 
  backing up all databases on virt0 | 
  [production] | 
            
  | 17:46 | 
  <Ryan_Lane> | 
  stopping rabbitmq on virt0 | 
  [production] | 
            
  | 17:43 | 
  <Ryan_Lane> | 
  disabling openstack services on the virt infrastructure, that is | 
  [production] | 
            
  | 17:43 | 
  <Ryan_Lane> | 
  disabling all openstack services | 
  [production] | 
            
  | 17:42 | 
  <Ryan_Lane> | 
  disabling OpenStackManager and LdapAuthentication on labsconsole | 
  [production] | 
            
  | 17:37 | 
  <Ryan_Lane> | 
  disabling puppet on all virt nodes | 
  [production] | 
            
  | 17:33 | 
  <paravoid> | 
  rolled back sq51's config; depooled sq51 (10' ago) | 
  [production] | 
            
  | 17:19 | 
  <paravoid> | 
  modified sq51 to serve origs from swift; added sq51 back to the pool | 
  [production] | 
            
  | 16:45 | 
  <paravoid> | 
  depooling sq51 backend | 
  [production] | 
            
  | 15:55 | 
  <paravoid> | 
  upgraded swift on ms-be12 | 
  [production] | 
            
  | 15:54 | 
  <paravoid> | 
  upgraded swift on ms-be5 | 
  [production] | 
            
  | 15:47 | 
  <paravoid> | 
  upgraded swift on ms-be11 | 
  [production] | 
            
  | 14:51 | 
  <paravoid> | 
  upgraded swift on ms-be3 | 
  [production] | 
            
  | 14:40 | 
  <paravoid> | 
  upgraded swift on ms-be4 | 
  [production] |