| 2010-09-07
      
      § | 
    
  | 21:41 | <Ryan_Lane> | set sq68 to false in pybal's bits configuration | [production] | 
            
  | 19:56 | <catrope> | synchronized php-1.5/languages/messages/MessagesFrr.php  'r72545' | [production] | 
            
  | 19:46 | <apergos> | restarted en wiki full history (with prefetch this time). | [production] | 
            
  | 19:29 | <robh> | synchronized php-1.5/wmf-config/mc.php  'adding in srv258-289 into memcached spares, all test online' | [production] | 
            
  | 18:47 | <Ryan_Lane> | split pdf1-3 and mobile1-3 away from misc cluster in ganglia; made two new clusters (pdf and mobile) | [production] | 
            
  | 18:33 | <apergos> | starting xml full history dumps stage (paralellized) for en wik on snapshot2 in screen session from /backups-testing | [production] | 
            
  | 17:38 | <Ryan_Lane> | changed password for Utilisateur:Bot de paille on frwiki | [production] | 
            
  | 14:37 | <mark> | Powercycled mobile1 | [production] | 
            
  | 05:13 | <apergos> | started meta-current phase of xml dumps for en wiki on snapshot2 in screen session | [production] | 
            
  | 03:39 | <Tim> | installed packet loss monitor on locke | [production] | 
            
  
    | 2010-09-03
      
      § | 
    
  | 23:44 | <Ryan_Lane> | added philippe to the wmffundraising alias | [production] | 
            
  | 18:56 | <RobH> | srv123 pulled, wiping.  also all memcached in updated pool test fine. | [production] | 
            
  | 18:56 | <robh> | synchronized php-1.5/wmf-config/mc.php  'removing srv123 from memcached due to decommissioning' | [production] | 
            
  | 18:38 | <RobH> | rt#54 db16 seems to not be throwing errors... installs ok, passed all VTS testing and memtesting.  reinstalled the OS | [production] | 
            
  | 11:45 | <tomaszf> | leaving lighttpd in non ipv6 mode till currently open sockets clear | [production] | 
            
  | 10:39 | <tomaszf> | disabling ipv6 for lighttpd on dataset1 as its not working post upgrade | [production] | 
            
  | 10:20 | <tomaszf> | upgrading to ubuntu 10.04 LTS on dataset1 | [production] | 
            
  | 01:44 | <Ryan_Lane> | added pre-commit and post-commit hooks to puppet's svn to check syntax on pre-commit, and svn up on post-commit | [production] | 
            
  
    | 2010-09-02
      
      § | 
    
  | 23:09 | <Ryan_Lane> | restarted mysql on db10 | [production] | 
            
  | 23:09 | <Ryan_Lane> | set expire_logs_days to 14 on db10 | [production] | 
            
  | 23:04 | <Ryan_Lane> | fixed password issue for debian-sys-maint on db10 | [production] | 
            
  | 22:56 | <Ryan_Lane> | commented out skip_slave_start in the my.cnf on db10 | [production] | 
            
  | 22:54 | <Ryan_Lane> | restarted db10 (specifically the mysql database) | [production] | 
            
  | 22:54 | <Ryan_Lane> | restarted db10 | [production] | 
            
  | 19:38 | <tfinc> | synchronized php-1.5/wmf-config/CommonSettings.php  'adding new thursday test banner to whitelist' | [production] | 
            
  | 16:53 | <RobH> | reinstalling storage2 | [production] | 
            
  | 16:09 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/SpecialPrefSwitch.php  'Attempt to fix DB error' | [production] | 
            
  | 16:06 | <RobH> | db16 is running VTS, DO NOT TOUCH THIS SERVER | [production] | 
            
  | 15:51 | <RobH> | db34 unracked, going back for repair with Oracle Field Engineer. | [production] | 
            
  | 15:02 | <RobH> | ignore errrors from db16, db28, db34.  Today is sun server troubleshooting day. | [production] | 
            
  | 14:58 | <RobH> | db28 is not in rotation, taking it down to continue hardware testing/repair | [production] | 
            
  | 14:25 | <RobH> | srv206 back online and in cluster. | [production] | 
            
  | 14:25 | <robh> | synchronized php-1.5/wmf-config/mc.php  'moved srv206 from down to spare' | [production] | 
            
  | 13:49 | <RobH> | srv206 offline, looking into it. | [production] |