| 2013-01-23
      
      § | 
    
  | 19:28 | <aaron> | rebuilt wikiversions.cdb and synchronized wikiversions files: Switched all non-wikipedias to 1.21-wmf8 | [production] | 
            
  | 19:07 | <RobH> | db61 coming offline, and going into otrs sandbox service | [production] | 
            
  | 19:05 | <RobH> | authdns-update | [production] | 
            
  | 18:03 | <binasher> | amended wikiadmin grants on s7 to allow new jobrunners access to centralauth | [production] | 
            
  | 17:44 | <paravoid> | running authdns-update - a bunch of cleanups, [[rev:3861|r3861]]:3867 | [production] | 
            
  | 15:09 | <mark> | Added missing pim interface ae0.400 statement on cr2-pmtpa which prevented eqiad purges from reaching pmtpa | [production] | 
            
  | 04:11 | <tstarling> | synchronized wmf-config/db-eqiad.php  'cleanup, added pmtpa host/IP mappings' | [production] | 
            
  | 04:09 | <tstarling> | synchronized wmf-config/db-pmtpa.php  'back to read/write mode (with eqiad master)' | [production] | 
            
  | 02:49 | <LocalisationUpdate> | completed (1.21wmf8) at Wed Jan 23 02:49:23 UTC 2013 | [production] | 
            
  | 02:26 | <LocalisationUpdate> | completed (1.21wmf7) at Wed Jan 23 02:26:04 UTC 2013 | [production] | 
            
  | 02:23 | <Tim> | reloading all apaches | [production] | 
            
  | 02:19 | <Tim> | running sync-apache to fix stale eqiad apache configuration | [production] | 
            
  | 02:14 | <Tim> | added eqiad apaches to /etc/dsh/group/apaches | [production] | 
            
  | 01:23 | <aaron> | synchronized php-1.21wmf8/includes/api/ApiUpload.php  'deployed 2a4ad3e32cf4f6814554ebcf09ac0250546a2549' | [production] | 
            
  | 01:22 | <aaron> | synchronized php-1.21wmf7/includes/api/ApiUpload.php  'deployed 2a4ad3e32cf4f6814554ebcf09ac0250546a2549' | [production] | 
            
  | 01:07 | <Tim> | ran scap-1 on tmh1001 and tmh1002 | [production] | 
            
  | 01:05 | <Tim> | added tmh1001 and tmh1002 to mediawiki-installation | [production] | 
            
  
    | 2013-01-22
      
      § | 
    
  | 23:58 | <Tim> | on db1006: fixing permissions for root@fenari | [production] | 
            
  | 23:54 | <Tim> | wikiadmin permissions for eqiad were broken, attempting to use CIDR notation, fixing them on all masters | [production] | 
            
  | 23:04 | <Ryan_Lane> | Alex Monk's echo changes to OpenStackManager deployed on labsconsole | [production] | 
            
  | 23:04 | <Ryan_Lane> | echo enabled on labsconsole | [production] | 
            
  | 23:03 | <Ryan_Lane> | labsconsole upgraded to 1.21wmf8 | [production] | 
            
  | 22:38 | <Tim> | on neon: cleaned up /var/lib/puppet/clientbucket/* since it was out of disk space | [production] | 
            
  | 22:29 | <catrope> | synchronized php-1.21wmf8/resources/jquery/jquery.client.js  '8a06466e1697f58717e92532437be642a796d5a1' | [production] | 
            
  | 22:29 | <catrope> | synchronized php-1.21wmf7/resources/jquery/jquery.client.js  '8a06466e1697f58717e92532437be642a796d5a1' | [production] | 
            
  | 21:55 | <cmjohnson1> | msw-a7-eqiad power was accidently removed while switch was being moved | [production] | 
            
  | 21:45 | <^demon> | gerrit restarting. don't panic. | [production] | 
            
  | 20:26 | <catrope> | synchronized php-1.21wmf8/includes/api/ApiQueryImageInfo.php  'Fix API error in imageinfo' | [production] | 
            
  | 20:26 | <catrope> | synchronized php-1.21wmf7/includes/api/ApiQueryImageInfo.php  'Fix API error in imageinfo' | [production] | 
            
  | 20:12 | <AaronSchulz> | Recompiled textvc on all hosts | [production] | 
            
  | 19:37 | <Ryan_Lane> | upgrading labsconsole to 1.21wmf8 | [production] | 
            
  | 19:03 | <catrope> | synchronized wmf-config/CommonSettings.php  'Disabling ReaderFeedback' | [production] | 
            
  | 19:03 | <catrope> | synchronized wmf-config/CommonSettings.php  'Disabling ReaderFeedback' | [production] | 
            
  | 18:59 | <mutante> | restarted pdns on ns1, failed during update | [production] | 
            
  | 18:58 | <mutante> | DNS update - adding ca./ja. planet to zirconium for testing new planet | [production] | 
            
  | 18:51 | <mutante> | DNS update - deleting wikimania2006/2007 | [production] | 
            
  | 18:14 | <root> | synchronized wmf-config/db-eqiad.php  'eqiad read-write!' | [production] | 
            
  | 17:59 | <notpeter> | moving jobrunning to eqiad | [production] | 
            
  | 17:54 | <aaron> | synchronized php-1.21wmf7/includes/Block.php  'deployed b0d5ff9bfb09f15ccbab6849564511d372ecd8d4' | [production] | 
            
  | 17:46 | <notpeter> | authdns-update | [production] | 
            
  | 17:42 | <Jeff_Green> | db1013 wiped, reimaged, powered off, and un-monitored | [production] | 
            
  | 17:40 | <asher> | synchronized wmf-config/db-pmtpa.php  'setting pmtpa to readonly' | [production] | 
            
  | 17:14 | <notpeter> | dns svn repo on sockpuppet has changes staged and checked in, not yet deployed on dobson | [production] | 
            
  | 17:05 | <notpeter> | staging cname switch for pmtpa/eqiad dbs on sockpuppet | [production] | 
            
  | 15:22 | <mark> | Changed AS14907->AS43821 routing | [production] | 
            
  | 12:26 | <^demon> | restarted ircecho on manganese | [production] | 
            
  | 10:30 | <Tim> | also replaced /etc/sudoers on fenari and hume. Hume needs a puppet change which will follow shortly | [production] | 
            
  | 10:18 | <hashar> | gallium : restarted puppet | [production] | 
            
  | 10:14 | <hashar> | jenkins updated all plugins and restarting | [production] | 
            
  | 10:09 | <Tim> | on searchidx2: replaced /etc/sudoers with the distro default so that /etc/sudoers.d/* from puppet can take effect | [production] |