| 2011-08-10
      
      § | 
    
  | 20:49 | <preilly> | fixing mobile view | [production] | 
            
  | 20:49 | <reedy> | synchronized wmf-config/CommonSettings.php  'Temporarily disabling MObileFrontend' | [production] | 
            
  | 20:49 | <preilly> | synchronizing Wikimedia installation... Revision: 93960: | [production] | 
            
  | 20:48 | <RoanKattouw> | MobileFrontend broke, serving mobile pages to everyone. Fixing | [production] | 
            
  | 20:43 | <preilly> | pushing fix for Mobile Frontend device detection | [production] | 
            
  | 20:43 | <preilly> | synchronizing Wikimedia installation... Revision: 93960: | [production] | 
            
  | 19:02 | <preilly> | synchronizing Wikimedia installation... Revision: 93960: | [production] | 
            
  | 19:02 | <preilly> | updating mobile frontend extension on production | [production] | 
            
  | 18:41 | <notpeter> | upgrading squid and squid-frontend on amssq49 | [production] | 
            
  | 17:01 | <mark> | Deployed new HTCP loss ganglia monitoring plugin on ganglia aggregator squids in pmtpa | [production] | 
            
  | 16:32 | <mutante> | sq51 & sq62 - dist-upgrade (incl. kernel) / reboot | [production] | 
            
  | 16:01 | <mutante> | sq51,sq62: power cycle, squid clean | [production] | 
            
  | 02:18 | <LocalisationUpdate> | completed at Wed Aug 10 02:21:00 UTC 2011 | [production] | 
            
  | 00:12 | <maplebed> | trying out a change to puppet's nagios config generation - moving service checks to one file per host. | [production] | 
            
  
    | 2011-08-08
      
      § | 
    
  | 20:41 | <binasher> | added self to auth portion of nagios cgi.cfg | [production] | 
            
  | 20:35 | <binasher> | powered off knsq30, disabling in nagios | [production] | 
            
  | 19:23 | <reedy> | synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js  'blanking file' | [production] | 
            
  | 18:51 | <tfinc> | synchronized wmf-config/CommonSettings.php  'Disabling WikimediaMobile client side redirect' | [production] | 
            
  | 18:49 | <tfinc> | synchronized wmf-config/InitialiseSettings.php  'Disabling mobile frontend' | [production] | 
            
  | 18:06 | <reedy> | synchronized wmf-config/CommonSettings.php  'Bug 25957 - Update' | [production] | 
            
  | 17:57 | <reedy> | synchronized wmf-config/flaggedrevs.php  '[[bugzilla:29911|bug 29911]]' | [production] | 
            
  | 17:46 | <reedy> | synchronized php-1.17/languages/Names.php  '[[rev:94069|r94069]]' | [production] | 
            
  | 17:27 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'fix language code for [[bugzilla:30045|bug 30045]]' | [production] | 
            
  | 17:15 | <Ryan_Lane> | RobH is removing the sep11.wikipedia.org subdomain redirect. It didn't post to the SAL, and he can't get onto IRC | [production] | 
            
  | 13:53 | <mark> | Fixed IPv6 router advertisements on cr1-sdtpa | [production] | 
            
  | 11:57 | <mark> | Killed squid on knsq30 | [production] | 
            
  | 11:52 | <mutante> | restarted squid on knsq30 | [production] | 
            
  | 11:29 | <mark> | Restored routing of pmtpa - esams path | [production] | 
            
  | 11:19 | <mark> | Restarted lsearchd on search7 | [production] | 
            
  | 02:18 | <LocalisationUpdate> | completed at Mon Aug  8 02:21:05 UTC 2011 | [production] | 
            
  
    | 2011-08-06
      
      § | 
    
  | 15:52 | <mutante> | amssq41 - and while we're on it: dist-upgrade incl. kernel and reboot. did not include squid package, that was downgraded on Apr 20 | [production] | 
            
  | 15:25 | <mutante> | amssq41 - power back up after it went down, clean squid cache | [production] | 
            
  | 13:07 | <RobH> | puppet run on spence is via screen on root user | [production] | 
            
  | 13:05 | <RobH> | spence is running puppet to update with new nagios info that won't contain the facilities fasle reports.  if it fails to work, ~/rob_puppet_services.cfg is on spence | [production] | 
            
  | 13:03 | <RobH> | nagios is being regenerated, i hope this works! | [production] | 
            
  | 12:23 | <RobH> | srv169 depooled, needs work and to be restored to service | [production] | 
            
  | 12:10 | <root> | synchronized wmf-config/mc.php  'replacing srv169 in active mc role due to its constant flapping' | [production] | 
            
  | 06:46 | <RobH> | emptied the /var/log/exim4/paniclog on project2, formey, lily, mchenry, sockpuppet, williams, srv196.  All errors were quite old due to various network blips over time. | [production] | 
            
  | 02:19 | <LocalisationUpdate> | completed at Sat Aug  6 02:21:34 UTC 2011 | [production] | 
            
  | 01:49 | <Reedy> | srv169 seems to be flapping | [production] | 
            
  | 01:48 | <Reedy> | Ganglia seems brokened "There was an error collecting ganglia data (127.0.0.1:8654): fsockopen error: Connection refused" | [production] |