| 
      
        2011-01-04
      
      §
     | 
  
    
  | 23:23 | 
  <Ryan_Lane> | 
  moving payments2 from internal to squid vlan | 
  [production] | 
            
  | 23:23 | 
  <Ryan_Lane> | 
  restarting network services on payments2 | 
  [production] | 
            
  | 23:18 | 
  <Ryan_Lane> | 
  restarting nginx on payments1 | 
  [production] | 
            
  | 23:12 | 
  <Ryan_Lane> | 
  switching IP address info for payments2 | 
  [production] | 
            
  | 23:11 | 
  <Ryan_Lane> | 
  moved payments1 from the internal to the squid vlan | 
  [production] | 
            
  | 22:12 | 
  <Ryan_Lane> | 
  restarting network services on payments1 | 
  [production] | 
            
  | 21:52 | 
  <Ryan_Lane> | 
  changing IP address for payments1 | 
  [production] | 
            
  | 20:07 | 
  <robh> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'changed tenwiki anon edit block to anon page creation block' | 
  [production] | 
            
  | 16:53 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'Remove hack disallowing anon edits on tenwiki - moved to its proper place in InitialiseSettings.php' | 
  [production] | 
            
  | 16:52 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'Disallow anon edits on tenwiki' | 
  [production] | 
            
  | 16:50 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'bug 26554 - Set $wgAutoConfirmAge to 0 on tenwiki' | 
  [production] | 
            
  | 16:43 | 
  <Ryan_Lane> | 
  removing login.tesla from DNS | 
  [production] | 
            
  | 15:42 | 
  <apergos> | 
  snapshot1 removed from  mediawiki-installation for now til set up again | 
  [production] | 
            
  | 15:35 | 
  <Ryan_Lane> | 
  dropping nova database on nova-controller.tesla, letting nova recreate it | 
  [production] | 
            
  | 15:29 | 
  <Ryan_Lane> | 
  changing ldap schema on nova-controller.tesla to newer (v2) openstack schema | 
  [production] | 
            
  | 14:24 | 
  <robh> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  | 
  [production] | 
            
  | 14:12 | 
  <Ryan_Lane> | 
  powercycling mobile3 | 
  [production] | 
            
  | 12:17 | 
  <RoanKattouw> | 
  srv182, srv183 seem to have gone into swap death | 
  [production] | 
            
  | 12:11 | 
  <catrope> | 
  synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.i18n.php  'r79572' | 
  [production] | 
            
  | 09:54 | 
  <apergos> | 
  restarted lsearcdh on search7 (in theory the one host in the search-pool3 ) to see if that helps | 
  [production] | 
            
  | 06:28 | 
  <root> | 
  synchronizing Wikimedia installation... Revision: 79563:  | 
  [production] | 
            
  | 06:27 | 
  <tstarling> | 
  synchronizing Wikimedia installation... Revision: 79563:  | 
  [production] | 
            
  | 06:26 | 
  <Tim> | 
  svn up/scap to r79563 for clickjacking defences | 
  [production] | 
            
  | 02:47 | 
  <Ryan_Lane> | 
  disabling the puppetmaster service, and enabling the opendj service on nova-controller.tesla | 
  [production] | 
            
  | 02:45 | 
  <Ryan_Lane> | 
  clearing mediawiki database for nova-controller.tesla, and clearing out most of the LDAP data | 
  [production] | 
            
  | 02:31 | 
  <Ryan_Lane> | 
  rebooting nova-*.tesla | 
  [production] | 
            
  | 02:21 | 
  <Ryan_Lane> | 
  updating nova-* on nova-controller.tesla | 
  [production] | 
            
  
    | 
      
        2011-01-03
      
      §
     | 
  
    
  | 23:19 | 
  <Ryan_Lane> | 
  deleting login.tesla vm | 
  [production] | 
            
  | 23:05 | 
  <Ryan_Lane> | 
  adding reedy account, putting samreed account back in, and ensuring the ssh key is absent for samreed | 
  [production] | 
            
  | 23:00 | 
  <Ryan_Lane> | 
  adding rfaulk to storage3 | 
  [production] | 
            
  | 22:58 | 
  <Ryan_Lane> | 
  renaming samreed to reedy in admins.pp | 
  [production] | 
            
  | 22:51 | 
  <Ryan_Lane> | 
  adding demon to kaulen | 
  [production] | 
            
  | 22:51 | 
  <Ryan_Lane> | 
  giving robla access to modify-ldap-user, add-ldap-user, and delete-ldap-user on formey | 
  [production] | 
            
  | 18:19 | 
  <midom> | 
  synchronized php-1.5/wmf-config/db.php  'db15 going live' | 
  [production] | 
            
  | 17:44 | 
  <apergos> | 
  reboot snapshot1 for install of lucid | 
  [production] | 
            
  | 16:18 | 
  <robh> | 
  synchronized php-1.5/wmf-config/mc.php  'replacing srv266 since its down' | 
  [production] | 
            
  | 16:16 | 
  <RobH> | 
  rebooting srv266 | 
  [production] | 
            
  | 14:34 | 
  <mark> | 
  Site should be stable again | 
  [production] | 
            
  | 14:08 | 
  <mark> | 
  Depooled text squids again | 
  [production] | 
            
  | 13:59 | 
  <mark> | 
  Adding text.esams squids into the bits.esams LVS pool | 
  [production] | 
            
  | 13:43 | 
  <mark> | 
  Killed pybal on amslvs1 | 
  [production] |