| 2012-06-27
      
      § | 
    
  | 23:50 | <K4-7131> | sync'd payments cluster to 592e0a5ba195 | [production] | 
            
  | 23:43 | <preilly> | synchronized wmf-config/InitialiseSettings.php  'add rule for mediawiki' | [production] | 
            
  | 22:37 | <K4-7131> | sync'd payments cluster to 7e9072c2d571c | [production] | 
            
  | 22:23 | <binasher> | temporarily pulling srv211 from pybal | [production] | 
            
  | 21:56 | <RobH> | mw1102 has no nic0, rather than troubleshoot it for a long time, reinstall!  (rt 3058) | [production] | 
            
  | 21:01 | <preilly> | synchronized php-1.20wmf6/extensions/MobileFrontend  'fix CSRF' | [production] | 
            
  | 21:00 | <preilly> | synchronized php-1.20wmf5/extensions/MobileFrontend  'fix CSRF' | [production] | 
            
  | 20:55 | <RobH> | db1003 back online, replaced mgmt cable and mgmt is working now as well | [production] | 
            
  | 20:51 | <LeslieCarr> | rebooting srv266 as it is unresponsive | [production] | 
            
  | 20:44 | <RobH> | db1003 mgmt issue due to bad cable, system booting back up, replacing mgmt cable | [production] | 
            
  | 20:35 | <RobH> | clean mysql shutdown, db1003 now offline | [production] | 
            
  | 20:33 | <RobH> | db1003 mgmt is not responsible, I need to remove power and reboot.  confirmed iwth asher this is an s3 slave and can do a short downtime without issues | [production] | 
            
  | 20:32 | <maxsem> | synchronized php-1.20wmf5/extensions/MobileFrontend/  'MF fixes and logging' | [production] | 
            
  | 20:29 | <maxsem> | synchronized php-1.20wmf6/extensions/MobileFrontend/  'MF fixes and logging' | [production] | 
            
  | 19:56 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'disable LastModified and LastModified/E3Experiment' | [production] | 
            
  | 19:54 | <reedy> | synchronized php-1.20wmf6/maintenance/runJobs.php | [production] | 
            
  | 19:53 | <reedy> | synchronized php-1.20wmf5/maintenance/runJobs.php | [production] | 
            
  | 19:32 | <reedy> | synchronized php-1.20wmf6/extensions/Translate/specials/SpecialLanguageStats.php | [production] | 
            
  | 19:19 | <reedy> | synchronized php-1.20wmf6/extensions/Translate/ | [production] | 
            
  | 19:06 | <reedy> | synchronized php-1.20wmf6/extensions/Translate/ | [production] | 
            
  | 19:03 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: meta back to wmf6, not cause of translate issues | [production] | 
            
  | 18:53 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: closed to 1.20wmf6 | [production] | 
            
  | 18:51 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia wikis to 1.20wmf6 | [production] | 
            
  | 18:48 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikiversity to 1.20wmf6 | [production] | 
            
  | 18:47 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiquote to 1.20wmf6 | [production] | 
            
  | 18:47 | <Jeff_Green> | added several mobile hostnames to DNS for RT #2996 | [production] | 
            
  | 18:46 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikibooks to 1.20wmf6 | [production] | 
            
  | 18:44 | <aaron> | rebuilt wikiversions.cdb and synchronized wikiversions files: Moved metawiki back to 1.20wmf5 | [production] | 
            
  | 18:41 | <K4-713> | synchronized payments cluster to fundraising/1.20 de0256084a | [production] | 
            
  | 18:33 | <aaron> | rebuilt wikiversions.cdb and synchronized wikiversions files: Moved special wikis to php-1.20wmf6 | [production] | 
            
  | 18:29 | <aaron> | rebuilt wikiversions.cdb and synchronized wikiversions files: Moved en(wikibooks|wikinews|wikiquote|wikisource|wikiversity|wiktionary) to 1.20wmf6 | [production] | 
            
  | 18:05 | <RobH> | cp1017 memory replaced | [production] | 
            
  | 17:52 | <RobH> | cp1017 is offline due to memory error.  replacement memory on site, pulling system for swap | [production] | 
            
  | 17:46 | <preilly> | synchronized php-1.20wmf6/extensions/ZeroRatedMobileAccess  'update for landing page' | [production] | 
            
  | 17:46 | <asher> | synchronized wmf-config/mc.php  'disabling wgMemCachedPersistent; lowering wgMemCachedTimeout to 2x client default from 30x default' | [production] | 
            
  | 17:45 | <preilly> | synchronized php-1.20wmf5/extensions/ZeroRatedMobileAccess  'update for landing page' | [production] | 
            
  | 17:44 | <preilly> | synchronized php-1.20wmf4/extensions/ZeroRatedMobileAccess  'update for landing page' | [production] | 
            
  | 17:20 | <LeslieCarr> | restarting apache2 on srv258 | [production] | 
            
  | 17:11 | <maplebed> | powercycled srv270 | [production] | 
            
  | 17:11 | <mutante> | powercycling srv277 (had to, frozen console) | [production] | 
            
  | 17:06 | <LeslieCarr> | rebooting srv287 | [production] | 
            
  | 17:05 | <Ryan_Lane> | rebooting srv280 | [production] | 
            
  | 17:03 | <mutante> | powercycling srv280 | [production] | 
            
  | 17:01 | <paravoid> | rebooting srv264, swapdeath | [production] | 
            
  | 16:52 | <mark> | Rebooting srv279, swapdeath | [production] | 
            
  | 16:52 | <paravoid> | rebooting srv275, swapdeath | [production] | 
            
  | 16:50 | <paravoid> | rebooting srv258, swapdeath | [production] | 
            
  | 16:43 | <reedy> | synchronized php-1.20wmf5/extensions/ArticleFeedbackv5/ | [production] |