| 2011-08-08
      
      § | 
    
  | 20:41 | <binasher> | added self to auth portion of nagios cgi.cfg | [production] | 
            
  | 20:35 | <binasher> | powered off knsq30, disabling in nagios | [production] | 
            
  | 19:23 | <reedy> | synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js  'blanking file' | [production] | 
            
  | 18:51 | <tfinc> | synchronized wmf-config/CommonSettings.php  'Disabling WikimediaMobile client side redirect' | [production] | 
            
  | 18:49 | <tfinc> | synchronized wmf-config/InitialiseSettings.php  'Disabling mobile frontend' | [production] | 
            
  | 18:06 | <reedy> | synchronized wmf-config/CommonSettings.php  'Bug 25957 - Update' | [production] | 
            
  | 17:57 | <reedy> | synchronized wmf-config/flaggedrevs.php  '[[bugzilla:29911|bug 29911]]' | [production] | 
            
  | 17:46 | <reedy> | synchronized php-1.17/languages/Names.php  '[[rev:94069|r94069]]' | [production] | 
            
  | 17:27 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'fix language code for [[bugzilla:30045|bug 30045]]' | [production] | 
            
  | 17:15 | <Ryan_Lane> | RobH is removing the sep11.wikipedia.org subdomain redirect. It didn't post to the SAL, and he can't get onto IRC | [production] | 
            
  | 13:53 | <mark> | Fixed IPv6 router advertisements on cr1-sdtpa | [production] | 
            
  | 11:57 | <mark> | Killed squid on knsq30 | [production] | 
            
  | 11:52 | <mutante> | restarted squid on knsq30 | [production] | 
            
  | 11:29 | <mark> | Restored routing of pmtpa - esams path | [production] | 
            
  | 11:19 | <mark> | Restarted lsearchd on search7 | [production] | 
            
  | 02:18 | <LocalisationUpdate> | completed at Mon Aug  8 02:21:05 UTC 2011 | [production] | 
            
  
    | 2011-08-06
      
      § | 
    
  | 15:52 | <mutante> | amssq41 - and while we're on it: dist-upgrade incl. kernel and reboot. did not include squid package, that was downgraded on Apr 20 | [production] | 
            
  | 15:25 | <mutante> | amssq41 - power back up after it went down, clean squid cache | [production] | 
            
  | 13:07 | <RobH> | puppet run on spence is via screen on root user | [production] | 
            
  | 13:05 | <RobH> | spence is running puppet to update with new nagios info that won't contain the facilities fasle reports.  if it fails to work, ~/rob_puppet_services.cfg is on spence | [production] | 
            
  | 13:03 | <RobH> | nagios is being regenerated, i hope this works! | [production] | 
            
  | 12:23 | <RobH> | srv169 depooled, needs work and to be restored to service | [production] | 
            
  | 12:10 | <root> | synchronized wmf-config/mc.php  'replacing srv169 in active mc role due to its constant flapping' | [production] | 
            
  | 06:46 | <RobH> | emptied the /var/log/exim4/paniclog on project2, formey, lily, mchenry, sockpuppet, williams, srv196.  All errors were quite old due to various network blips over time. | [production] | 
            
  | 02:19 | <LocalisationUpdate> | completed at Sat Aug  6 02:21:34 UTC 2011 | [production] | 
            
  | 01:49 | <Reedy> | srv169 seems to be flapping | [production] | 
            
  | 01:48 | <Reedy> | Ganglia seems brokened "There was an error collecting ganglia data (127.0.0.1:8654): fsockopen error: Connection refused" | [production] | 
            
  
    | 2011-08-05
      
      § | 
    
  | 23:50 | <tfinc> | removed knsq30 from LVS esams rotation | [production] | 
            
  | 23:36 | <Reedy> | Even though Asher pulled knsq30 from rotation, it seems to be still serving users | [production] | 
            
  | 21:39 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'more whitespace' | [production] | 
            
  | 21:33 | <reedy> | synchronized wmf-config/CommonSettings.php  'whitespace' | [production] | 
            
  | 21:32 | <^demon> | fixed permissions on new hook scripts so they're executable | [production] | 
            
  | 21:28 | <^demon> | deployed new svn pre-commit hook framework | [production] | 
            
  | 21:01 | <binasher> | pulled knsq30 from all frontend squids | [production] | 
            
  | 20:44 | <root> | synchronized wmf-config/db.php  'returning db18 to s7' | [production] | 
            
  | 19:02 | <apergos> | power cycled dataset2, it was unresponsive to pings and form mgmt console | [production] | 
            
  | 18:24 | <binasher> | squid frontend deploy - no mobile redirect for android 3 (tablets) | [production] | 
            
  | 18:15 | <Reedy> | running namespaceDupes on plwikibooks | [production] | 
            
  | 18:14 | <reedy> | synchronized wmf-config/InitialiseSettings.php  '[[bugzilla:30164|bug 30164]] Change namespaces configuration - pl.wikibooks' | [production] | 
            
  | 18:00 | <reedy> | synchronized wmf-config/abusefilter.php  'Tidy up some redundant userrights definitions' | [production] | 
            
  | 17:56 | <Reedy> | That sync was also [[bugzilla:30207|bug 30207]] Enable AbuseFilter on Russian Wikinews | [production] | 
            
  | 17:56 | <reedy> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 17:55 | <reedy> | synchronized wmf-config/abusefilter.php  '[[bugzilla:30207|bug 30207]] Enable AbuseFilter on Russian Wikinews' | [production] |