| 2012-01-12
      
      § | 
    
  | 21:54 | <Ryan_Lane> | relabeled port at virt0 | [production] | 
            
  | 21:54 | <Ryan_Lane> | moved new virt0 from squid vlan to public-services2 | [production] | 
            
  | 21:43 | <Ryan_Lane> | rebuilding mobile2 as virt0 | [production] | 
            
  | 21:43 | <Ryan_Lane> | Adding back mgmt info for mobile1, changing mobile2 to virt0 | [production] | 
            
  | 21:11 | <Ryan_Lane> | rebuilding mobile1 as virt0 | [production] | 
            
  | 21:08 | <Ryan_Lane> | renaming mobile1 to virt0 | [production] | 
            
  | 20:54 | <binasher> | installing percona-toolkit on few remaining hardy dbs | [production] | 
            
  | 20:26 | <cmjohnson1> | shutting down srv178-189 for decommissioning | [production] | 
            
  | 20:14 | <binasher> | granted the "process" priv to nagios@localhost on all production db clusters | [production] | 
            
  | 20:07 | <reedy> | synchronized php-1.18/includes/specials/SpecialSearch.php  '[[rev:108751|r108751]]' | [production] | 
            
  | 20:07 | <LeslieCarr> | reassigning ports on asw-b-sdtpa | [production] | 
            
  | 17:00 | <notpeter> | stop sodium to do manual reinstall | [production] | 
            
  | 16:33 | <RobH> | adjusting all power strip humidity sensor 2 (floor level) to 12% humidity, as the center rack has the proper levels, floor levels always are low in humidity. | [production] | 
            
  | 16:04 | <mutante> | starting nagios-nrpe-server on ALL via dsh to speed up nagios recovery | [production] | 
            
  | 15:33 | <mutante> | starting nagios-nrpe-server on srv's via dsh | [production] | 
            
  | 02:04 | <LocalisationUpdate> | completed (1.18) at Thu Jan 12 02:04:31 UTC 2012 | [production] | 
            
  | 00:48 | <preilly> | pushing quick fix for special random | [production] | 
            
  | 00:48 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php  'update to mobile frontend to fix random link' | [production] | 
            
  | 00:41 | <LeslieCarr> | added ganglia1002 and ganglia1001 to dns | [production] | 
            
  
    | 2012-01-11
      
      § | 
    
  | 23:18 | <RobH> | searchidx1001 offline and powered down until replacement memory arrives (2012-01-13) rt 2208 | [production] | 
            
  | 22:56 | <RobH> | poking searchidx1001 for memory error | [production] | 
            
  | 22:45 | <RobH> | mw1108 online and ready for install per rt2253 | [production] | 
            
  | 22:42 | <RobH> | mw1099 repaired, ready for os install per rt2252 | [production] | 
            
  | 22:39 | <RobH> | mw1081 ready for install rt2251 | [production] | 
            
  | 22:32 | <RobH> | no its not ;] | [production] | 
            
  | 22:16 | <Reedy> | lists.wikimedia.org is down | [production] | 
            
  | 21:53 | <reedy> | synchronized php-1.18/includes/api/  '[[rev:108683|r108683]]' | [production] | 
            
  | 21:36 | <RobH> | psw1-eqiad mgmt connected | [production] | 
            
  | 21:24 | <RobH> | leslie is handling the ganglia not starting back up issue even though i caused it to die, yay me | [production] | 
            
  | 21:22 | <RobH> | updated dns for neon/cobalt to ganglia1001/1002 | [production] | 
            
  | 21:17 | <RobH> | ganglia offline for a moment, sorry folks | [production] | 
            
  | 21:17 | <catrope> | synchronizing Wikimedia installation... : Deploying MoodBar changes | [production] | 
            
  | 21:16 | <RobH> | i just took nickel offline by mistake | [production] | 
            
  | 20:58 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Change shorturl preefix default' | [production] | 
            
  | 20:57 | <reedy> | synchronized php-1.18/extensions/ShortUrl/  '[[rev:108680|r108680]]' | [production] | 
            
  | 20:50 | <RoanKattouw> | Applying MoodBar schema changes (index addition and column addition) on all wikis | [production] | 
            
  | 20:44 | <catrope> | synchronized php-1.18/extensions/ArticleFeedbackv5/  'Updating AFTv5 to trunk staet' | [production] | 
            
  | 20:14 | <Jeff_Green> | adjusted firewall rules on payments* to restore ganglia reporting since we switched to nickel | [production] | 
            
  | 20:11 | <RoanKattouw> | Created AFTv5 tables on testwiki | [production] | 
            
  | 20:09 | <catrope> | synchronized php-1.18/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js  '[[rev:108666|r108666]]' | [production] | 
            
  | 19:53 | <cmjohnson1> | shutting down srv191 for new install | [production] | 
            
  | 19:52 | <cmjohnson1> | replaced HDD srv191 | [production] | 
            
  | 19:47 | <catrope> | synchronized php-1.18/resources/startup.js  'touch' | [production] | 
            
  | 19:46 | <catrope> | synchronized wmf-config/InitialiseSettings.php  'Enable AFTv5 on testwiki' | [production] |