| 
      
        2012-09-28
      
      §
     | 
  
    
  | 18:55 | 
  <reedy> | 
  synchronized wmf-config/  | 
  [production] | 
            
  | 18:38 | 
  <reedy> | 
  synchronized wmf-config/  | 
  [production] | 
            
  | 18:04 | 
  <binasher> | 
  stopping mysql on db1047 for upgrades | 
  [production] | 
            
  | 16:19 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Set wgCopyUploadProxy and enable copyupload on testwiki' | 
  [production] | 
            
  | 16:17 | 
  <reedy> | 
  synchronized php-1.20wmf12/includes/upload/UploadFromUrl.php  | 
  [production] | 
            
  | 16:16 | 
  <reedy> | 
  synchronized php-1.20wmf12/includes/DefaultSettings.php  | 
  [production] | 
            
  | 15:46 | 
  <RobH> | 
  done with db62.  it was in the ubuntu installer when I took it over, so its just sitting now | 
  [production] | 
            
  | 14:53 | 
  <RobH> | 
  db62 being pulled for drac7 work | 
  [production] | 
            
  | 14:39 | 
  <Jeff_Green> | 
  rebooting aluminium | 
  [production] | 
            
  | 13:11 | 
  <mark> | 
  Moved BGP transit sessions to AS1257 from cr2-eqiad (over equinix exchange) to cr1-eqiad (dedicated link) | 
  [production] | 
            
  | 02:26 | 
  <LocalisationUpdate> | 
  completed (1.20wmf12) at Fri Sep 28 02:26:27 UTC 2012 | 
  [production] | 
            
  
    | 
      
        2012-09-27
      
      §
     | 
  
    
  | 23:13 | 
  <olivneh> | 
  synchronized php-1.20wmf12/extensions/Vector/modules/ext.vector.footerCleanup.css  | 
  [production] | 
            
  | 22:50 | 
  <olivneh> | 
  synchronized wmf-config  | 
  [production] | 
            
  | 22:05 | 
  <olivneh> | 
  synchronized php-1.20wmf12/extensions/E3Experiments  | 
  [production] | 
            
  | 22:00 | 
  <bsitu> | 
  Finished syncing Wikimedia installation... : Update PageTriage and ArticleFeedbackv5 to master | 
  [production] | 
            
  | 21:32 | 
  <bsitu> | 
  Started syncing Wikimedia installation... : Update PageTriage and ArticleFeedbackv5 to master | 
  [production] | 
            
  | 20:48 | 
  <kaldari> | 
  synchronized wmf-config/InitialiseSettings.php  'turning off moodbar on testwiki' | 
  [production] | 
            
  | 20:09 | 
  <RobH> | 
  authdns-update for new misc servers mgmt in eqiad | 
  [production] | 
            
  | 19:10 | 
  <cmjohnson1> | 
  search32 going down to swap cpu's (dell request) | 
  [production] | 
            
  | 18:58 | 
  <reedy> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files:  | 
  [production] | 
            
  | 17:42 | 
  <cmjohnson1> | 
  bring db62 down to swap disk controller card per ct/asher | 
  [production] | 
            
  | 17:16 | 
  <demon> | 
  synchronized wmf-config/throttle.php  'Syncing I8c44415b: Account creation throttle for ptwikiversity event' | 
  [production] | 
            
  | 16:42 | 
  <cmjohnson1> | 
  ms-be6 powering down | 
  [production] | 
            
  | 16:37 | 
  <RobH> | 
  cp1031 mainboard replaced per rt3614, will need reinstall | 
  [production] | 
            
  | 15:59 | 
  <RobH> | 
  mc1006 memory repaired rt3613 | 
  [production] | 
            
  | 15:54 | 
  <RobH> | 
  mc1002 repaired rt3612 | 
  [production] | 
            
  | 15:33 | 
  <RobH> | 
  mc1002 (rt3612) & mc1006 (rt3613) offline for memory repair/swap | 
  [production] | 
            
  | 15:21 | 
  <RobH> | 
  authdns-update for wikimediafoundation.info | 
  [production] | 
            
  | 15:06 | 
  <Jeff_Green> | 
  indium dist-upgrade & reboot | 
  [production] | 
            
  | 13:58 | 
  <Jeff_Green> | 
  dist-upgrade & reboot grosley | 
  [production] | 
            
  | 10:55 | 
  <apergos> | 
  ignore ms-be6 messages, I'm trying to get into the bleeping lsi raid util | 
  [production] | 
            
  | 04:04 | 
  <olivneh> | 
  synchronized php-1.20wmf12/extensions/E3Experiments/experiments/postEditFeedback.js  | 
  [production] | 
            
  | 02:36 | 
  <Jeff_Green> | 
  powercycle fenari, it's nonresponsive even via drac terminal | 
  [production] | 
            
  
    | 
      
        2012-09-26
      
      §
     | 
  
    
  | 20:42 | 
  <RobH> | 
  mc1011 drac fixed | 
  [production] | 
            
  | 20:39 | 
  <RobH> | 
  mc1009, 1010 drac fixed | 
  [production] | 
            
  | 20:30 | 
  <RobH> | 
  cp1031 will remain offline until replacement parts arrive tomorrow rt3614 | 
  [production] | 
            
  | 18:44 | 
  <RobH> | 
  correction rt 3613 | 
  [production] | 
            
  | 18:43 | 
  <RobH> | 
  mc1006 testing done, single bad dimm, support ticket filed rt3614 | 
  [production] | 
            
  | 18:16 | 
  <reedy> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: 285 remaining wikis to 1.20wmf12 | 
  [production] | 
            
  | 18:06 | 
  <RobH> | 
  mc1006 offline for memory troubleshooting per rt 3613 | 
  [production] | 
            
  | 16:52 | 
  <RobH> | 
  mc1002 offline, memory bad per rt3612, new memory will arrive tomorrow | 
  [production] | 
            
  | 16:14 | 
  <RobH> | 
  carbon rebooting for console redirection issues, any eqiad based installs will fail during this downtime | 
  [production] | 
            
  | 15:57 | 
  <Jeff_Green> | 
  dist-upgrade & reboot most payments boxes | 
  [production] | 
            
  | 15:21 | 
  <RobH> | 
  bast1001 link speed fixed, done working on it | 
  [production] | 
            
  | 15:19 | 
  <RobH> | 
  troubleshooting bast1001 link speed per 3414 | 
  [production] | 
            
  | 15:12 | 
  <cmjohnson1> | 
  sq37 has fatal error...powering down to replace disk controller card | 
  [production] | 
            
  | 13:56 | 
  <cmjohnson1> | 
  ms-be8 powercycling | 
  [production] | 
            
  | 02:48 | 
  <LocalisationUpdate> | 
  completed (1.20wmf11) at Wed Sep 26 02:48:22 UTC 2012 | 
  [production] | 
            
  | 02:23 | 
  <LocalisationUpdate> | 
  completed (1.20wmf12) at Wed Sep 26 02:23:25 UTC 2012 | 
  [production] | 
            
  | 01:00 | 
  <mutante> | 
  reinstalling cp1030 with precise | 
  [production] |