| 2011-12-16
      
      § | 
    
  | 13:01 | <mark> | Found lvs5 and lvs6 with offload-gro enabled, even though it's set disabled in /etc/network/interfaces... corrected | [production] | 
            
  | 09:21 | <apergos> | restarted lighthttpd on ds2, it had stopped (and why didn't nagios tell us? ) | [production] | 
            
  | 08:38 | <mutante> | spence - had killed additional notifications.cgi and history.cgi procs, waited 5 minutes, load went down a lot, restarting nagios | [production] | 
            
  | 08:23 | <mutante> | spence - almost unusable, Nagios notifications.cgi and history.cgi use a lot of memory, stopping Nagios, watching swap | [production] | 
            
  | 08:15 | <mutante> | spence slow again, side-note: tried to use "sar" to investigate but "Please check if data collecting is enabled in /etc/default/sysstat" (want to?) | [production] | 
            
  | 07:54 | <nikerabbit> | synchronized php-1.18/extensions/WebFonts/resources/ext.webfonts.js  'JS fix [[rev:106418|r106418]]' | [production] | 
            
  | 07:09 | <apergos> | thumbs cleaner awake for the day | [production] | 
            
  | 01:57 | <LocalisationUpdate> | failed (1.18) at Fri Dec 16 02:00:14 UTC 2011 | [production] | 
            
  
    | 2011-12-15
      
      § | 
    
  | 23:19 | <LeslieCarr> | pushing rule to planet.wikimedia.org which should redirect all https to http | [production] | 
            
  | 23:00 | <LeslieCarr> | puppetized planet.wikimedia.org on singer | [production] | 
            
  | 22:41 | <LeslieCarr> | removing https support from planet.wikimedia.org | [production] | 
            
  | 21:43 | <awjrichards> | synchronized php/extensions/LandingCheck/SpecialLandingCheck.php  '[[rev:106377|r106377]]' | [production] | 
            
  | 21:42 | <awjrichards> | synchronized php/extensions/LandingCheck/LandingCheck.php  '[[rev:106377|r106377]]' | [production] | 
            
  | 21:34 | <awjrichards> | synchronized php/extensions/ContributionTracking/ContributionTracking_body.php  '[[rev:106375|r106375]]' | [production] | 
            
  | 21:34 | <awjrichards> | synchronized php/extensions/ContributionTracking/ContributionTracking.processor.php  '[[rev:106375|r106375]]' | [production] | 
            
  | 21:33 | <awjrichards> | synchronized php/extensions/ContributionTracking/ContributionTracking.php  '[[rev:106375|r106375]]' | [production] | 
            
  | 20:21 | <K4-713> | synchronized payments cluster to [[rev:106360|r106360]] | [production] | 
            
  | 19:16 | <LeslieCarr> | removed mw* and virt* yaml files from stafford in order to clear up broken files and make puppet run again | [production] | 
            
  | 18:59 | <LocalisationUpdate> | completed (1.18) at Thu Dec 15 19:02:58 UTC 2011 | [production] | 
            
  | 18:58 | <RobH> | cp1018 also offline, stealing cables from it for testing | [production] | 
            
  | 18:52 | <reedy> | synchronized php-1.18/includes/Block.php  'Testing fix for [[bugzilla:33101|bug 33101]]' | [production] | 
            
  | 18:43 | <catrope> | synchronized php-1.18/extensions/LocalisationUpdate/  '[[rev:106352|r106352]]' | [production] | 
            
  | 18:25 | <RobH> | es1002 and cp1019 offline for harddisk controller testing | [production] | 
            
  | 18:07 | <RoanKattouw> | Running LU /again/ to hopefully fix issues | [production] | 
            
  | 16:31 | <mark> | Fixed database entries in william's exim.conf | [production] | 
            
  | 16:24 | <catrope> | synchronized wmf-config/InitialiseSettings.php  'Disabling Contest extension because of XSS' | [production] | 
            
  | 15:31 | <apergos> | thumb cleaner to bed for the night | [production] | 
            
  | 14:59 | <LocalisationUpdate> | completed (1.18) at Thu Dec 15 15:02:05 UTC 2011 | [production] | 
            
  | 14:52 | <RoanKattouw> | Running l10nupdate by hand | [production] | 
            
  | 14:37 | <RoanKattouw> | That InitialiseSettings.php change enabled ArticleFeedback (v4) on en_labswikimedia | [production] | 
            
  | 14:37 | <catrope> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 14:28 | <catrope> | synchronized wmf-config/InitialiseSettings.php  'Disable ReaderFeedback on en_labswikimedia' | [production] | 
            
  | 14:26 | <catrope> | synchronized php-1.18/extensions/ArticleFeedbackv5/  'Updating AFTv5 to trunk staet' | [production] | 
            
  | 13:56 | <Reedy> | FixExtLinksProtocolRelative run now finished on all wikis | [production] | 
            
  | 07:04 | <apergos> | thumb cleaner awake for the day | [production] | 
            
  | 04:00 | <Tim> | restarted all job runners to get Roan's jobs-loop.sh change | [production] | 
            
  | 02:13 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/  'weekly update to mobile frontend' | [production] | 
            
  | 01:59 | <LocalisationUpdate> | failed (1.18) at Thu Dec 15 02:02:13 UTC 2011 | [production] | 
            
  | 01:53 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/  'weekly update to mobile frontend' | [production] | 
            
  | 00:28 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/  'weekly update to mobile frontend' | [production] | 
            
  | 00:25 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/  'weekly update to mobile frontend' | [production] | 
            
  
    | 2011-12-14
      
      § | 
    
  | 23:55 | <catrope> | synchronized php-1.18/maintenance/jobs-loop.sh  '[[rev:106284|r106284]]' | [production] | 
            
  | 23:54 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/  'weekly update to mobile frontend' | [production] | 
            
  | 23:48 | <catrope> | synchronized php-1.18/resources/startup.js  'touch' | [production] | 
            
  | 23:47 | <catrope> | synchronized wmf-config/InitialiseSettings.php  'Enable ClickTracking on en_labswikimedia so AFTv5 will work' | [production] | 
            
  | 23:29 | <catrope> | synchronized wmf-config/InitialiseSettings.php  'Enable AFTv5 on en_labswikimedia' | [production] | 
            
  | 23:22 | <catrope> | synchronizing Wikimedia installation... : Deploying MoodBar changes for feedback dashboard. Along for the ride: VisualEditor changes by Neil and a Badtitle bug fix by Tim | [production] | 
            
  | 23:14 | <RoanKattouw> | Created AFTv5 tables on en_labswikimedia and enwiki | [production] | 
            
  | 23:01 | <catrope> | synchronized wmf-config/CommonSettings.php  'Add trigger code for AFTv5' | [production] | 
            
  | 22:59 | <binasher> | restarted nagios with enable_environment_macros = 0 | [production] |