| 
      
        2011-04-29
      
      §
     | 
  
    
  | 23:27 | 
  <^demon> | 
  all of that was to deploy GoogleNewsSitemap to testwiki for the enwikinews folks. | 
  [production] | 
            
  | 23:23 | 
  <demon> | 
  synchronized php-1.17/wmf-config/CommonSettings.php  | 
  [production] | 
            
  | 23:22 | 
  <demon> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  | 
  [production] | 
            
  | 23:18 | 
  <demon> | 
  synchronizing Wikimedia installation... Revision: 87149: r87149 | 
  [production] | 
            
  | 19:43 | 
  <catrope> | 
  synchronized php-1.17/wmf-config/CommonSettings.php  'Fix config bug that caused tracking to fail for AFT, and bump tracking version' | 
  [production] | 
            
  | 19:18 | 
  <mark> | 
  powercycled ersch | 
  [production] | 
            
  | 15:40 | 
  <RobH> | 
  typo asw-b8 | 
  [production] | 
            
  | 15:39 | 
  <RobH> | 
  swapping msw1-eqiad fan tray with asw-a8-eqiad fan tray | 
  [production] | 
            
  | 05:58 | 
  <midom> | 
  synchronized php-1.17/wmf-config/db.php  'adding srv170 after hw reset' | 
  [production] | 
            
  | 05:55 | 
  <midom> | 
  synchronized php-1.17/wmf-config/db.php  | 
  [production] | 
            
  
    | 
      
        2011-04-28
      
      §
     | 
  
    
  | 23:46 | 
  <Tim> | 
  doing hard reset of alsted, it's not sending ssh version strings and not allowing logins by serial | 
  [production] | 
            
  | 22:17 | 
  <pdhanda> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  'Disabled option to allow blocked users to edit their talk pages on itwiki. See Bug 9073' | 
  [production] | 
            
  | 21:02 | 
  <Ryan_Lane> | 
  created Ambassador-announce-l list | 
  [production] | 
            
  | 20:52 | 
  <richcole1> | 
  replacing memory in srv281 | 
  [production] | 
            
  | 20:49 | 
  <RobH> | 
  srv281 shutdown for memory replacement | 
  [production] | 
            
  | 20:41 | 
  <pdhanda> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  'Enable by default, option to enable blocked users to edit their own talk page' | 
  [production] | 
            
  | 20:34 | 
  <richcole1> | 
  replacing bad drive in srv284 | 
  [production] | 
            
  | 20:08 | 
  <Ryan_Lane> | 
  powercycling srv281 | 
  [production] | 
            
  | 20:03 | 
  <RobH> | 
  ms4 is powered down, was already offline and unresponsive to console. | 
  [production] | 
            
  | 18:19 | 
  <demon> | 
  synchronizing Wikimedia installation... Revision: 87091: r87090, r87091 | 
  [production] | 
            
  | 18:02 | 
  <mark> | 
  Restarted apache on srv235, probably corrupted apc cache | 
  [production] | 
            
  | 17:57 | 
  <awjrichards> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  'Re-enabling EmailCapture on enwiki' | 
  [production] | 
            
  | 17:54 | 
  <awjrichards> | 
  synchronizing Wikimedia installation... Revision: 87084:  | 
  [production] | 
            
  | 17:47 | 
  <awjrichards> | 
  synchronized php-1.17/wmf-config/CommonSettings.php  'Preparing conf for ArticleFeedback with way less click tracking to prevent cluster breakage' | 
  [production] | 
            
  | 17:40 | 
  <RobH> | 
  etherpad.proxy to 000-etherpad.proxy on hooper apache2 enabled, should fix | 
  [production] | 
            
  | 17:34 | 
  <RobH> | 
  hooper vhost for testblog back online | 
  [production] | 
            
  | 17:17 | 
  <RobH> | 
  rolled back testblog vhost from hooper | 
  [production] | 
            
  | 17:01 | 
  <RobH> | 
  testblog enabled for initial setup on hooper without disrupting etherpad service (on either of its url redirects) | 
  [production] | 
            
  | 16:56 | 
  <RobH> | 
  updating etherpad.proxy apache config to include eiximenis serveralias and enabling the testblog vhost | 
  [production] | 
            
  | 16:48 | 
  <Reedy> | 
  Manually pinged CR update script, had got stuck - 2 revisions. Let me know if it does it again | 
  [production] | 
            
  | 16:21 | 
  <catrope> | 
  synchronizing Wikimedia installation... Revision: 87085:  | 
  [production] | 
            
  | 16:21 | 
  <RoanKattouw> | 
  Rolling back ArticleFeedback deployment | 
  [production] | 
            
  | 16:11 | 
  <RoanKattouw> | 
  API cluster was briefly dead just now after enabling EmailCapture, disabling it seems to be bringing it back | 
  [production] | 
            
  | 16:09 | 
  <catrope> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  'Disable EmailCapture, maybe this will bring our API Apaches back' | 
  [production] | 
            
  | 16:04 | 
  <awjrichards> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  'Enabling EmailCapture on enwiki' | 
  [production] | 
            
  | 16:02 | 
  <RoanKattouw> | 
  (that was for the ArticleFeedback deployment) | 
  [production] | 
            
  | 15:58 | 
  <awjrichards> | 
  synchronizing Wikimedia installation... Revision: 87083:  | 
  [production] | 
            
  | 14:08 | 
  <nelson> | 
  Upped the max_connections for rsyncd from 2 to 5 across the test cluster. | 
  [production] | 
            
  | 13:56 | 
  <nelson> | 
  Upped the concurrency of swift-object-replicator from default of 1 to 5 | 
  [production] | 
            
  | 13:38 | 
  <mark> | 
  Restored I/O scheduler deadline for device /dev/sda on alsted | 
  [production] | 
            
  | 13:34 | 
  <mark> | 
  Temporarily set I/O scheduler cfq for device /dev/sda on alsted | 
  [production] | 
            
  | 12:29 | 
  <mark> | 
  powercycled harmon | 
  [production] | 
            
  | 01:00 | 
  <awjr> | 
  upgraded dev.civicrm.wikimedia.org to CiviCRM 3.4.0 | 
  [production] |