| 
      
        2013-01-23
      
      §
     | 
  
    
  | 23:57 | 
  <dzahn> | 
  gracefulled all apaches | 
  [production] | 
            
  | 22:53 | 
  <RobH> | 
  calcium is my testbed server, ignore its bitching | 
  [production] | 
            
  | 22:29 | 
  <mutante> | 
  starting NTP on mw1008 (Tick tock, on the clock But the party don't stop, no) | 
  [production] | 
            
  | 22:20 | 
  <paravoid> | 
  netapp: nas1001-b cf giveback | 
  [production] | 
            
  | 22:10 | 
  <RobH> | 
  authdns-update correcting camera fqdn | 
  [production] | 
            
  | 22:00 | 
  <Ryan_Lane> | 
  restarting pdns on ns0 | 
  [production] | 
            
  | 21:56 | 
  <cmjohnson1> | 
  authdns update adding mgmt ip's to zone files for 10 misc servers in eqiad | 
  [production] | 
            
  | 21:30 | 
  <Ryan_Lane> | 
  restarting opendj on virt0 | 
  [production] | 
            
  | 21:12 | 
  <mutante> | 
  restarting pdns on ns1 | 
  [production] | 
            
  | 21:03 | 
  <RobH> | 
  authdns-update to push db61 into sandbox | 
  [production] | 
            
  | 20:56 | 
  <RobH> | 
  swapped port 3/39 on csw1-sdtpa from vlan 2 to 102 | 
  [production] | 
            
  | 20:24 | 
  <mutante> | 
  removing mw1072 (hardware fail, RT-4381) from "apaches" dsh group | 
  [production] | 
            
  | 20:22 | 
  <dzahn> | 
  gracefulled all apaches | 
  [production] | 
            
  | 19:43 | 
  <mutante> | 
  puppet signing mw1085 (was sitting there like a fresh install) | 
  [production] | 
            
  | 19:41 | 
  <RobH> | 
  stopping puppet on brewster cuz i wanna locally hack and reboot 1 time on db61 without partman auto kicking in | 
  [production] | 
            
  | 19:28 | 
  <aaron> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: Switched all non-wikipedias to 1.21-wmf8 | 
  [production] | 
            
  | 19:07 | 
  <RobH> | 
  db61 coming offline, and going into otrs sandbox service | 
  [production] | 
            
  | 19:05 | 
  <RobH> | 
  authdns-update | 
  [production] | 
            
  | 18:03 | 
  <binasher> | 
  amended wikiadmin grants on s7 to allow new jobrunners access to centralauth | 
  [production] | 
            
  | 17:44 | 
  <paravoid> | 
  running authdns-update - a bunch of cleanups, [[rev:3861|r3861]]:3867 | 
  [production] | 
            
  | 15:09 | 
  <mark> | 
  Added missing pim interface ae0.400 statement on cr2-pmtpa which prevented eqiad purges from reaching pmtpa | 
  [production] | 
            
  | 04:11 | 
  <tstarling> | 
  synchronized wmf-config/db-eqiad.php  'cleanup, added pmtpa host/IP mappings' | 
  [production] | 
            
  | 04:09 | 
  <tstarling> | 
  synchronized wmf-config/db-pmtpa.php  'back to read/write mode (with eqiad master)' | 
  [production] | 
            
  | 02:49 | 
  <LocalisationUpdate> | 
  completed (1.21wmf8) at Wed Jan 23 02:49:23 UTC 2013 | 
  [production] | 
            
  | 02:26 | 
  <LocalisationUpdate> | 
  completed (1.21wmf7) at Wed Jan 23 02:26:04 UTC 2013 | 
  [production] | 
            
  | 02:23 | 
  <Tim> | 
  reloading all apaches | 
  [production] | 
            
  | 02:19 | 
  <Tim> | 
  running sync-apache to fix stale eqiad apache configuration | 
  [production] | 
            
  | 02:14 | 
  <Tim> | 
  added eqiad apaches to /etc/dsh/group/apaches | 
  [production] | 
            
  | 01:23 | 
  <aaron> | 
  synchronized php-1.21wmf8/includes/api/ApiUpload.php  'deployed 2a4ad3e32cf4f6814554ebcf09ac0250546a2549' | 
  [production] | 
            
  | 01:22 | 
  <aaron> | 
  synchronized php-1.21wmf7/includes/api/ApiUpload.php  'deployed 2a4ad3e32cf4f6814554ebcf09ac0250546a2549' | 
  [production] | 
            
  | 01:07 | 
  <Tim> | 
  ran scap-1 on tmh1001 and tmh1002 | 
  [production] | 
            
  | 01:05 | 
  <Tim> | 
  added tmh1001 and tmh1002 to mediawiki-installation | 
  [production] | 
            
  
    | 
      
        2013-01-22
      
      §
     | 
  
    
  | 23:58 | 
  <Tim> | 
  on db1006: fixing permissions for root@fenari | 
  [production] | 
            
  | 23:54 | 
  <Tim> | 
  wikiadmin permissions for eqiad were broken, attempting to use CIDR notation, fixing them on all masters | 
  [production] | 
            
  | 23:04 | 
  <Ryan_Lane> | 
  Alex Monk's echo changes to OpenStackManager deployed on labsconsole | 
  [production] | 
            
  | 23:04 | 
  <Ryan_Lane> | 
  echo enabled on labsconsole | 
  [production] | 
            
  | 23:03 | 
  <Ryan_Lane> | 
  labsconsole upgraded to 1.21wmf8 | 
  [production] | 
            
  | 22:38 | 
  <Tim> | 
  on neon: cleaned up /var/lib/puppet/clientbucket/* since it was out of disk space | 
  [production] | 
            
  | 22:29 | 
  <catrope> | 
  synchronized php-1.21wmf8/resources/jquery/jquery.client.js  '8a06466e1697f58717e92532437be642a796d5a1' | 
  [production] | 
            
  | 22:29 | 
  <catrope> | 
  synchronized php-1.21wmf7/resources/jquery/jquery.client.js  '8a06466e1697f58717e92532437be642a796d5a1' | 
  [production] | 
            
  | 21:55 | 
  <cmjohnson1> | 
  msw-a7-eqiad power was accidently removed while switch was being moved | 
  [production] |