| 2013-01-23
      
      § | 
    
  | 23:57 | <dzahn> | gracefulled all apaches | [production] | 
            
  | 22:53 | <RobH> | calcium is my testbed server, ignore its bitching | [production] | 
            
  | 22:29 | <mutante> | starting NTP on mw1008 (Tick tock, on the clock But the party don't stop, no) | [production] | 
            
  | 22:20 | <paravoid> | netapp: nas1001-b cf giveback | [production] | 
            
  | 22:10 | <RobH> | authdns-update correcting camera fqdn | [production] | 
            
  | 22:00 | <Ryan_Lane> | restarting pdns on ns0 | [production] | 
            
  | 21:56 | <cmjohnson1> | authdns update adding mgmt ip's to zone files for 10 misc servers in eqiad | [production] | 
            
  | 21:30 | <Ryan_Lane> | restarting opendj on virt0 | [production] | 
            
  | 21:12 | <mutante> | restarting pdns on ns1 | [production] | 
            
  | 21:03 | <RobH> | authdns-update to push db61 into sandbox | [production] | 
            
  | 20:56 | <RobH> | swapped port 3/39 on csw1-sdtpa from vlan 2 to 102 | [production] | 
            
  | 20:24 | <mutante> | removing mw1072 (hardware fail, RT-4381) from "apaches" dsh group | [production] | 
            
  | 20:22 | <dzahn> | gracefulled all apaches | [production] | 
            
  | 19:43 | <mutante> | puppet signing mw1085 (was sitting there like a fresh install) | [production] | 
            
  | 19:41 | <RobH> | stopping puppet on brewster cuz i wanna locally hack and reboot 1 time on db61 without partman auto kicking in | [production] | 
            
  | 19:28 | <aaron> | rebuilt wikiversions.cdb and synchronized wikiversions files: Switched all non-wikipedias to 1.21-wmf8 | [production] | 
            
  | 19:07 | <RobH> | db61 coming offline, and going into otrs sandbox service | [production] | 
            
  | 19:05 | <RobH> | authdns-update | [production] | 
            
  | 18:03 | <binasher> | amended wikiadmin grants on s7 to allow new jobrunners access to centralauth | [production] | 
            
  | 17:44 | <paravoid> | running authdns-update - a bunch of cleanups, [[rev:3861|r3861]]:3867 | [production] | 
            
  | 15:09 | <mark> | Added missing pim interface ae0.400 statement on cr2-pmtpa which prevented eqiad purges from reaching pmtpa | [production] | 
            
  | 04:11 | <tstarling> | synchronized wmf-config/db-eqiad.php  'cleanup, added pmtpa host/IP mappings' | [production] | 
            
  | 04:09 | <tstarling> | synchronized wmf-config/db-pmtpa.php  'back to read/write mode (with eqiad master)' | [production] | 
            
  | 02:49 | <LocalisationUpdate> | completed (1.21wmf8) at Wed Jan 23 02:49:23 UTC 2013 | [production] | 
            
  | 02:26 | <LocalisationUpdate> | completed (1.21wmf7) at Wed Jan 23 02:26:04 UTC 2013 | [production] | 
            
  | 02:23 | <Tim> | reloading all apaches | [production] | 
            
  | 02:19 | <Tim> | running sync-apache to fix stale eqiad apache configuration | [production] | 
            
  | 02:14 | <Tim> | added eqiad apaches to /etc/dsh/group/apaches | [production] | 
            
  | 01:23 | <aaron> | synchronized php-1.21wmf8/includes/api/ApiUpload.php  'deployed 2a4ad3e32cf4f6814554ebcf09ac0250546a2549' | [production] | 
            
  | 01:22 | <aaron> | synchronized php-1.21wmf7/includes/api/ApiUpload.php  'deployed 2a4ad3e32cf4f6814554ebcf09ac0250546a2549' | [production] | 
            
  | 01:07 | <Tim> | ran scap-1 on tmh1001 and tmh1002 | [production] | 
            
  | 01:05 | <Tim> | added tmh1001 and tmh1002 to mediawiki-installation | [production] | 
            
  
    | 2013-01-22
      
      § | 
    
  | 23:58 | <Tim> | on db1006: fixing permissions for root@fenari | [production] | 
            
  | 23:54 | <Tim> | wikiadmin permissions for eqiad were broken, attempting to use CIDR notation, fixing them on all masters | [production] | 
            
  | 23:04 | <Ryan_Lane> | Alex Monk's echo changes to OpenStackManager deployed on labsconsole | [production] | 
            
  | 23:04 | <Ryan_Lane> | echo enabled on labsconsole | [production] | 
            
  | 23:03 | <Ryan_Lane> | labsconsole upgraded to 1.21wmf8 | [production] | 
            
  | 22:38 | <Tim> | on neon: cleaned up /var/lib/puppet/clientbucket/* since it was out of disk space | [production] | 
            
  | 22:29 | <catrope> | synchronized php-1.21wmf8/resources/jquery/jquery.client.js  '8a06466e1697f58717e92532437be642a796d5a1' | [production] | 
            
  | 22:29 | <catrope> | synchronized php-1.21wmf7/resources/jquery/jquery.client.js  '8a06466e1697f58717e92532437be642a796d5a1' | [production] | 
            
  | 21:55 | <cmjohnson1> | msw-a7-eqiad power was accidently removed while switch was being moved | [production] |