| 
      
        2011-08-10
      
      §
     | 
  
    
  | 20:49 | 
  <preilly> | 
  fixing mobile view | 
  [production] | 
            
  | 20:49 | 
  <reedy> | 
  synchronized wmf-config/CommonSettings.php  'Temporarily disabling MObileFrontend' | 
  [production] | 
            
  | 20:49 | 
  <preilly> | 
  synchronizing Wikimedia installation... Revision: 93960:  | 
  [production] | 
            
  | 20:48 | 
  <RoanKattouw> | 
  MobileFrontend broke, serving mobile pages to everyone. Fixing | 
  [production] | 
            
  | 20:43 | 
  <preilly> | 
  pushing fix for Mobile Frontend device detection | 
  [production] | 
            
  | 20:43 | 
  <preilly> | 
  synchronizing Wikimedia installation... Revision: 93960:  | 
  [production] | 
            
  | 19:02 | 
  <preilly> | 
  synchronizing Wikimedia installation... Revision: 93960:  | 
  [production] | 
            
  | 19:02 | 
  <preilly> | 
  updating mobile frontend extension on production | 
  [production] | 
            
  | 18:41 | 
  <notpeter> | 
  upgrading squid and squid-frontend on amssq49 | 
  [production] | 
            
  | 17:01 | 
  <mark> | 
  Deployed new HTCP loss ganglia monitoring plugin on ganglia aggregator squids in pmtpa | 
  [production] | 
            
  | 16:32 | 
  <mutante> | 
  sq51 & sq62 - dist-upgrade (incl. kernel) / reboot | 
  [production] | 
            
  | 16:01 | 
  <mutante> | 
  sq51,sq62: power cycle, squid clean | 
  [production] | 
            
  | 02:18 | 
  <LocalisationUpdate> | 
  completed at Wed Aug 10 02:21:00 UTC 2011 | 
  [production] | 
            
  | 00:12 | 
  <maplebed> | 
  trying out a change to puppet's nagios config generation - moving service checks to one file per host. | 
  [production] | 
            
  
    | 
      
        2011-08-08
      
      §
     | 
  
    
  | 20:41 | 
  <binasher> | 
  added self to auth portion of nagios cgi.cfg | 
  [production] | 
            
  | 20:35 | 
  <binasher> | 
  powered off knsq30, disabling in nagios | 
  [production] | 
            
  | 19:23 | 
  <reedy> | 
  synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js  'blanking file' | 
  [production] | 
            
  | 18:51 | 
  <tfinc> | 
  synchronized wmf-config/CommonSettings.php  'Disabling WikimediaMobile client side redirect' | 
  [production] | 
            
  | 18:49 | 
  <tfinc> | 
  synchronized wmf-config/InitialiseSettings.php  'Disabling mobile frontend' | 
  [production] | 
            
  | 18:06 | 
  <reedy> | 
  synchronized wmf-config/CommonSettings.php  'Bug 25957 - Update' | 
  [production] | 
            
  | 17:57 | 
  <reedy> | 
  synchronized wmf-config/flaggedrevs.php  '[[bugzilla:29911|bug 29911]]' | 
  [production] | 
            
  | 17:46 | 
  <reedy> | 
  synchronized php-1.17/languages/Names.php  '[[rev:94069|r94069]]' | 
  [production] | 
            
  | 17:27 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'fix language code for [[bugzilla:30045|bug 30045]]' | 
  [production] | 
            
  | 17:15 | 
  <Ryan_Lane> | 
  RobH is removing the sep11.wikipedia.org subdomain redirect. It didn't post to the SAL, and he can't get onto IRC | 
  [production] | 
            
  | 13:53 | 
  <mark> | 
  Fixed IPv6 router advertisements on cr1-sdtpa | 
  [production] | 
            
  | 11:57 | 
  <mark> | 
  Killed squid on knsq30 | 
  [production] | 
            
  | 11:52 | 
  <mutante> | 
  restarted squid on knsq30 | 
  [production] | 
            
  | 11:29 | 
  <mark> | 
  Restored routing of pmtpa - esams path | 
  [production] | 
            
  | 11:19 | 
  <mark> | 
  Restarted lsearchd on search7 | 
  [production] | 
            
  | 02:18 | 
  <LocalisationUpdate> | 
  completed at Mon Aug  8 02:21:05 UTC 2011 | 
  [production] | 
            
  
    | 
      
        2011-08-06
      
      §
     | 
  
    
  | 15:52 | 
  <mutante> | 
  amssq41 - and while we're on it: dist-upgrade incl. kernel and reboot. did not include squid package, that was downgraded on Apr 20 | 
  [production] | 
            
  | 15:25 | 
  <mutante> | 
  amssq41 - power back up after it went down, clean squid cache | 
  [production] | 
            
  | 13:07 | 
  <RobH> | 
  puppet run on spence is via screen on root user | 
  [production] | 
            
  | 13:05 | 
  <RobH> | 
  spence is running puppet to update with new nagios info that won't contain the facilities fasle reports.  if it fails to work, ~/rob_puppet_services.cfg is on spence | 
  [production] | 
            
  | 13:03 | 
  <RobH> | 
  nagios is being regenerated, i hope this works! | 
  [production] | 
            
  | 12:23 | 
  <RobH> | 
  srv169 depooled, needs work and to be restored to service | 
  [production] | 
            
  | 12:10 | 
  <root> | 
  synchronized wmf-config/mc.php  'replacing srv169 in active mc role due to its constant flapping' | 
  [production] | 
            
  | 06:46 | 
  <RobH> | 
  emptied the /var/log/exim4/paniclog on project2, formey, lily, mchenry, sockpuppet, williams, srv196.  All errors were quite old due to various network blips over time. | 
  [production] | 
            
  | 02:19 | 
  <LocalisationUpdate> | 
  completed at Sat Aug  6 02:21:34 UTC 2011 | 
  [production] | 
            
  | 01:49 | 
  <Reedy> | 
  srv169 seems to be flapping | 
  [production] | 
            
  | 01:48 | 
  <Reedy> | 
  Ganglia seems brokened "There was an error collecting ganglia data (127.0.0.1:8654): fsockopen error: Connection refused" | 
  [production] |