| 2012-01-26
      
      § | 
    
  | 23:58 | <catrope> | synchronized php-1.18/extensions/MoodBar/  'Update MoodBar' | [production] | 
            
  | 23:56 | <catrope> | synchronized wmf-config/CommonSettings.php  'Adding $wgMoodbarConfig["feedbackDashboardUrl"]' | [production] | 
            
  | 23:54 | <reedy> | synchronized php-1.18/includes/specials/SpecialBlockList.php  '[[rev:110095|r110095]]' | [production] | 
            
  | 23:28 | <mark> | Deployed squid configs to all squids | [production] | 
            
  | 23:26 | <mark> | Deploying modified squid configs of modified squid config generator to text.knams | [production] | 
            
  | 22:14 | <RobH> | poking at puppet change breaking things on sockpuppet puppet runs | [production] | 
            
  | 21:33 | <^demon> | gallium: cleared a bunch of junk from /tmp | [production] | 
            
  | 21:12 | <Jeff_Green> | upgraded storage3 mysqld from 5.1.47 to mysql-at-facebook-[[rev:3753|r3753]] | [production] | 
            
  | 20:17 | <asher> | synchronized wmf-config/db.php  'db37 back in s7' | [production] | 
            
  | 20:10 | <Reedy> | Created "spoofuser" AntiSpoof table in the central auth database | [production] | 
            
  | 19:34 | <asher> | synchronized wmf-config/db.php  'pulling db37 from s7 for upgrades' | [production] | 
            
  | 19:24 | <RobH> | disregard any flapping by mw1001, its my script testbed | [production] | 
            
  | 18:04 | <RobH> | forcing puppet run on srv199 | [production] | 
            
  | 17:45 | <RobH> | shutting down srv199 for bios tinkering by chris | [production] | 
            
  | 02:37 | <RoanKattouw> | Started the udp2log process for the AFT logger manually on emery | [production] | 
            
  | 02:06 | <LocalisationUpdate> | completed (1.18) at Thu Jan 26 02:06:22 UTC 2012 | [production] | 
            
  | 01:01 | <asher> | synchronized wmf-config/db.php  'adding db32 to s1 at low weight, new enwiki snapshot host' | [production] | 
            
  | 00:33 | <asher> | synchronized wmf-config/db.php  'returning db18, now replicating heartbeat db' | [production] | 
            
  | 00:28 | <asher> | synchronized wmf-config/db.php  'temporarily pulling db18' | [production] | 
            
  | 00:11 | <pgehres> | re-enabling recurring donation module and processing in CiviCRM | [production] | 
            
  | 00:04 | <asher> | synchronized wmf-config/db.php  'adding db26 to s7' | [production] | 
            
  | 00:03 | <RobH> | shutting down srv151-srv186 per RT 2318 (confirmed not in pybal pools for apache or api) | [production] | 
            
  
    | 2012-01-25
      
      § | 
    
  | 23:59 | <binasher> | removed old external store apaches from pybal config | [production] | 
            
  | 23:50 | <asher> | synchronized wmf-config/db.php  'pulling db26' | [production] | 
            
  | 23:49 | <asher> | synchronized wmf-config/db.php  'adding db26 to s7' | [production] | 
            
  | 23:35 | <pgehres> | re-enabled queue consumption for payments through Jenkins | [production] | 
            
  | 23:35 | <pgehres> | awjr synchronized CiviCRM on aluminium to [[rev:1211|r1211]] | [production] | 
            
  | 23:05 | <awjrichards> | synchronized wmf-config/CommonSettings.php  'Checking if wmgDisplayFeedsInSidebar === false rather than true, since it defaults to true in the install file' | [production] | 
            
  | 22:50 | <awjrichards> | synchronizing Wikimedia installation... : Enabling FeaturedFeeds everywhere | [production] | 
            
  | 22:41 | <RobH> | updating dns for bellin/blondel db9/10 replacements | [production] | 
            
  | 22:32 | <awjrichards> | synchronizing Wikimedia installation... : Dark-deploying FeaturedFeeds | [production] | 
            
  | 22:07 | <awjrichards> | synchronized wmf-config/CommonSettings.php  'fixed spelling mistake fore FeaturedFeeds configuration' | [production] | 
            
  | 22:05 | <awjrichards> | synchronized wmf-config/CommonSettings.php  'Setting up FeaturedFeeds config; disabled by default' | [production] | 
            
  | 22:02 | <awjrichards> | synchronized wmf-config/InitialiseSettings.php  'Setting up FeaturedFeeds config; disabled by default' | [production] | 
            
  | 21:54 | <LeslieCarr> | deactivated selected-paths policy-statement on cr1-eqiad and cr2-eqiad | [production] | 
            
  | 21:23 | <Tim> | on srv197: compiled and installed a local version of wmerrors for segfault investigation | [production] | 
            
  | 21:22 | <binasher> | bits caches: running varnish param.set thread_pool_min, thread_pool_max, where min = 15000 / cores / 4 and max = 15000 / cores | [production] | 
            
  | 20:57 | <binasher> | running "varnishadm param.set thread_pool_max 1875" on mobile varnish servers | [production] | 
            
  | 20:05 | <Tim> | on srv197: temporarily disabled puppet and enabled core dumps in apache2.conf for segfault flood investigation | [production] | 
            
  | 19:57 | <Jeff_Green> | running dist-upgrade on payments* and silicon | [production] | 
            
  | 19:44 | <Tim> | updating TrustedXFF host list using fenari | [production] | 
            
  | 19:29 | <RobH> | dns update go! | [production] | 
            
  | 18:50 | <LeslieCarr> | restarted varnish on niobium | [production] | 
            
  | 18:49 | <RoanKattouw> | Restarted morebots | [production] | 
            
  | 13:33 | <demon> | synchronized wmf-config/CommonSettings.php  'Change IP address for bnwiki account creation throttle per [[bugzilla:33900|bug 33900]]' | [production] | 
            
  | 11:43 | <apergos> | formey oom (I guess), unresponsive from mgmt console, powercycling. | [production] | 
            
  | 06:07 | <pgehres> | disabled queue consumption of payments in jenkins until stuck message can be removed from queue | [production] |