| 
      
        2012-10-22
      
      §
     | 
  
    
  | 16:02 | 
  <logmsgbot_> | 
  reedy synchronized wmf-config/ | 
  [production] | 
            
  | 16:01 | 
  <logmsgbot_> | 
  reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor | 
  [production] | 
            
  | 15:48 | 
  <logmsgbot_> | 
  reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor | 
  [production] | 
            
  | 15:43 | 
  <notpeter> | 
  re-adding srv219 and srv220 to rendering pool | 
  [production] | 
            
  | 15:04 | 
  <logmsgbot_> | 
  reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor | 
  [production] | 
            
  | 13:32 | 
  <notpeter> | 
  removing srv219 and srv220 from rendering pool for upgarde to precise | 
  [production] | 
            
  | 09:35 | 
  <mark> | 
  Built new varnish 3.0.3plus~rc1-wm2 packages and inserted them into the precise-wikimedia APT repository | 
  [production] | 
            
  | 02:51 | 
  <logmsgbot_> | 
  LocalisationUpdate completed (1.21wmf1) at Mon Oct 22 02:51:32 UTC 2012 | 
  [production] | 
            
  | 02:31 | 
  <logmsgbot_> | 
  LocalisationUpdate completed (1.21wmf2) at Mon Oct 22 02:30:57 UTC 2012 | 
  [production] | 
            
  
    | 
      
        2012-10-21
      
      §
     | 
  
    
  | 22:50 | 
  <logmsgbot_> | 
  tstarling synchronized php-1.21wmf2/bin/ulimit4.sh | 
  [production] | 
            
  | 22:39 | 
  <Reedy> | 
  populateFilearchiveSha1.php finished on commonswiki | 
  [production] | 
            
  | 14:18 | 
  <Reedy> | 
  Running mwscript populateFilearchiveSha1.php commonswiki | 
  [production] | 
            
  | 02:49 | 
  <logmsgbot_> | 
  LocalisationUpdate completed (1.21wmf1) at Sun Oct 21 02:49:42 UTC 2012 | 
  [production] | 
            
  | 02:28 | 
  <logmsgbot_> | 
  LocalisationUpdate completed (1.21wmf2) at Sun Oct 21 02:28:50 UTC 2012 | 
  [production] | 
            
  | 01:35 | 
  <paravoid> | 
  powercycling srv219, swap-died while trying to get a core dump | 
  [production] | 
            
  | 01:34 | 
  <logmsgbot_> | 
  reedy synchronized wmf-config/CommonSettings.php  'wgMaxImageArea to 20MP' | 
  [production] | 
            
  | 01:33 | 
  <paravoid> | 
  powercycling srv220/srv224, swapdeath | 
  [production] | 
            
  
    | 
      
        2012-10-20
      
      §
     | 
  
    
  | 18:51 | 
  <Reedy> | 
  Ran populateFilearchiveSha1.php on metawiki | 
  [production] | 
            
  | 18:21 | 
  <paravoid> | 
  powercycling srv221 & srv222, both dead since ~5h ago | 
  [production] | 
            
  | 17:55 | 
  <RobH_> | 
  updated otrs motd per requst by otrs admins (it referenced a now closed request for comment) | 
  [production] | 
            
  | 17:42 | 
  <Reedy> | 
  Running foreachwiki cleanupUploadStash.php | 
  [production] | 
            
  | 03:13 | 
  <K4-713> | 
  updated the payments cluster to 97e1ba74a39376 | 
  [production] | 
            
  | 02:41 | 
  <logmsgbot_> | 
  LocalisationUpdate completed (1.21wmf1) at Sat Oct 20 02:41:10 UTC 2012 | 
  [production] | 
            
  | 02:21 | 
  <logmsgbot_> | 
  LocalisationUpdate completed (1.21wmf2) at Sat Oct 20 02:21:41 UTC 2012 | 
  [production] | 
            
  | 01:36 | 
  <K4-713> | 
  updated the payments cluster to bb30dab91a7afe862 | 
  [production] | 
            
  | 00:46 | 
  <mutante> | 
  upgrade Bugzilla to 4.0.8 (security patch) | 
  [production] | 
            
  
    | 
      
        2012-10-19
      
      §
     | 
  
    
  | 23:36 | 
  <logmsgbot_> | 
  reedy synchronized wmf-config/CommonSettings.php  'wgMaxImageArea to 17MP' | 
  [production] | 
            
  | 23:31 | 
  <logmsgbot_> | 
  reedy synchronized php-1.21wmf2/extensions/SwiftCloudFiles | 
  [production] | 
            
  | 23:30 | 
  <LeslieCarr> | 
  shutting down sfo-monitor1 | 
  [production] | 
            
  | 23:27 | 
  <logmsgbot_> | 
  reedy synchronized php-1.21wmf2/includes/api/ApiParse.php | 
  [production] | 
            
  | 23:15 | 
  <binasher> | 
  returning srv194/200 to apache lvs pool | 
  [production] | 
            
  | 22:50 | 
  <mutante> | 
  authdns-update, renaming travel-guide-lb to wikivoyage-lb | 
  [production] | 
            
  | 21:10 | 
  <binasher> | 
  pulled srv194 and srv200 from lvs for memc testing | 
  [production] | 
            
  | 21:02 | 
  <binasher> | 
  upgrading percona-toolkit to 2.1.5 | 
  [production] | 
            
  | 20:18 | 
  <logmsgbot_> | 
  reedy synchronized php-1.21wmf2/includes/api/ApiParse.php | 
  [production] | 
            
  | 19:33 | 
  <logmsgbot_> | 
  aaron synchronized php-1.21wmf2/includes/filebackend/FileBackendStore.php  'deployed 65e47037f6e9da2e81db2bc1374cdac375a82db5' | 
  [production] | 
            
  | 19:26 | 
  <paravoid> | 
  powercycling srv223 too, locked up, nothing in serial console | 
  [production] | 
            
  | 19:25 | 
  <paravoid> | 
  powercycling srv221, locked up, nothing in serial console | 
  [production] | 
            
  | 16:21 | 
  <RobH> | 
  msw-a2-eqiad back online | 
  [production] | 
            
  | 15:54 | 
  <RobH> | 
  msw-a2-eqiad is coming down for replacement per RT 3683.  all mgmt connectivity (only) will be lost to items in A2-EQIAD | 
  [production] | 
            
  | 14:43 | 
  <paravoid> | 
  putting ms-fe1 back into the pool | 
  [production] | 
            
  | 14:14 | 
  <logmsgbot_> | 
  reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki to 1.21wmf2 | 
  [production] | 
            
  | 13:59 | 
  <logmsgbot_> | 
  reedy rebuilt wikiversions.cdb and synchronized wikiversions files: metawiki to 1.21wmf2 | 
  [production] | 
            
  | 13:50 | 
  <cmjohnson1> | 
  shutting down msbe1001-1012 to remove ssd's and package for dell return | 
  [production] | 
            
  | 13:48 | 
  <logmsgbot_> | 
  reedy synchronized php-1.21wmf2/includes/EditPage.php | 
  [production] | 
            
  | 04:15 | 
  <logmsgbot_> | 
  tstarling synchronized php-1.21wmf2/includes/User.php | 
  [production] | 
            
  | 02:59 | 
  <logmsgbot_> | 
  tstarling synchronized php-1.21wmf2/extensions/AbuseFilter/AbuseFilter.class.php | 
  [production] | 
            
  | 02:57 | 
  <logmsgbot_> | 
  tstarling synchronized php-1.21wmf2/includes/User.php | 
  [production] | 
            
  | 02:47 | 
  <logmsgbot_> | 
  LocalisationUpdate completed (1.21wmf1) at Fri Oct 19 02:47:49 UTC 2012 | 
  [production] | 
            
  | 02:28 | 
  <logmsgbot_> | 
  LocalisationUpdate completed (1.21wmf2) at Fri Oct 19 02:27:57 UTC 2012 | 
  [production] |