| 2012-08-13
      
      § | 
    
  | 23:01 | <mutante> | killing nagios-wm process which isn't supposed to be on #wikimedia-tech anymore | [production] | 
            
  | 21:58 | <cmjohnson1> | running authdns-update for labsdb1 & 2 | [production] | 
            
  | 21:19 | <RoanKattouw> | ...piping result into ms7:/root/badownershipfiles | [production] | 
            
  | 21:18 | <RoanKattouw> | ms7# find /export/upload/wik*/*/{0,1,2,3,4,5,6,7,8,9,a,b,c,d,e,f,archive,math,temp,timeline} ! -user apache | [production] | 
            
  | 21:17 | <RoanKattouw> | Searching for files with wrong ownership on ms7 using | [production] | 
            
  | 21:13 | <RobH> | reinstalling virt1004 to see if the raid issue re-appears | [production] | 
            
  | 20:55 | <notpeter> | updating root auth keys file on ms7 by hand. | [production] | 
            
  | 20:36 | <catrope> | synchronized php-1.20wmf9/resources/jquery/jquery.tablesorter.js  'Deploy 1fafaef3aa157f49cbbf4b9cffb03544bcf72a08' | [production] | 
            
  | 20:04 | <preilly> | updating zero for custom banner colors | [production] | 
            
  | 20:03 | <preilly> | synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess  'update for landing page' | [production] | 
            
  | 20:03 | <preilly> | synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess  'update for landing page' | [production] | 
            
  | 19:42 | <preilly> | synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess  'update for landing page' | [production] | 
            
  | 19:42 | <preilly> | update to zero for banner text link | [production] | 
            
  | 19:41 | <preilly> | synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess  'update for landing page' | [production] | 
            
  | 19:36 | <preilly> | synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess  'update for landing page' | [production] | 
            
  | 19:32 | <RobH> | powering off analytics1011-1014 to remove the add on nics (disrupting installer) | [production] | 
            
  | 18:25 | <aaron> | rebuilt wikiversions.cdb and synchronized wikiversions files: Moved enwiki to 1.20wmf9 | [production] | 
            
  | 17:21 | <aaron> | synchronized wmf-config/filebackend.php  'Switched backend reads to swift for testwikis and mw.org' | [production] | 
            
  | 15:18 | <paravoid> | reprovisioning srv281 with a new fs layout | [production] | 
            
  | 13:05 | <mark> | Doing NDU of IOM3 shelf firmware on nas1001-a | [production] | 
            
  | 12:59 | <mark> | Doing NDU of IOM3 shelf firmware on nas1-a | [production] | 
            
  | 12:47 | <mark> | Setup NTP on all NetApps | [production] | 
            
  | 12:24 | <mark> | Restored SnapMirror relationship between nas1-a and nas1001-a, 70 days lagged, transfer initiated | [production] | 
            
  | 12:04 | <paravoid> | apt: include php-memcached 2.0.1-6~wmf+lucid2 @ lucid-wikimedia | [production] | 
            
  | 11:59 | <paravoid> | apt: include php5 5.3.10-1ubuntu3.2+wmf1 @ precise-wikimedia | [production] | 
            
  | 11:08 | <paravoid> | apt: remove wikidiff2/0.0.1wm1 (obsolete), copysrc php-wikidiff2 {lucid,precise}-wikimedia | [production] | 
            
  | 11:08 | <paravoid> | apt: remove php-wikidiff2/1.1.0-2 from lucid-wikimedia/universe (lucid-wikimedia/main has 1.1.2-1) | [production] | 
            
  | 10:49 | <apergos> | starting initial rsync from ms8 to ms10 (should take approximately forever) | [production] | 
            
  | 06:26 | <Tim> | uploaded czmq and zpubsub packages to brewster for Ori | [production] | 
            
  | 02:37 | <LocalisationUpdate> | completed (1.20wmf8) at Mon Aug 13 02:37:53 UTC 2012 | [production] | 
            
  | 02:20 | <LocalisationUpdate> | completed (1.20wmf9) at Mon Aug 13 02:20:27 UTC 2012 | [production] | 
            
  
    | 2012-08-10
      
      § | 
    
  | 21:54 | <Ryan_Lane> | live hacked nova on virt2 with parts of nova git commit 4584e552a653904c36cf04cb295a7bf09d2def28 | [production] | 
            
  | 21:23 | <preilly> | synchronized php-1.20wmf9/extensions/ZeroRatedMobileAccess  'update for landing page' | [production] | 
            
  | 21:22 | <preilly> | synchronized php-1.20wmf8/extensions/ZeroRatedMobileAccess  'update for landing page' | [production] | 
            
  | 21:21 | <preilly> | update to zero needed to support carrier testing | [production] | 
            
  | 21:21 | <preilly> | synchronized php-1.20wmf9/extensions/MobileFrontend  'update for zero' | [production] | 
            
  | 21:21 | <preilly> | update to zero needed to support carrier testing | [production] | 
            
  | 21:20 | <preilly> | synchronized php-1.20wmf8/extensions/MobileFrontend  'update for zero' | [production] | 
            
  | 18:42 | <Jeff_Green> | adding frack.eqiad.wmnet hosts to DNS (wmnet and 10..in-addr.arpa) | [production] | 
            
  | 18:11 | <K4-713> | Payments cluster updated to 133b2b6fb5 | [production] | 
            
  | 17:07 | <notpeter> | temporarily stopping puppet on brewster | [production] | 
            
  | 16:43 | <cmjohnson1> | enabling pxe boot for 10GB Nic cards mc1-16 | [production] | 
            
  | 16:32 | <mark> | Started OnTap upgrade to 8.1 on nas1-a and nas1-b | [production] | 
            
  | 14:39 | <mark> | OnTap upgrade on nas1001-a and -b complete | [production] | 
            
  | 14:11 | <mark> | Started OnTap upgrade to 8.1 on nas1001-b | [production] |