| 2011-09-30
      
      § | 
    
  | 16:13 | <Ryan_Lane> | pooling ssl3003 | [production] | 
            
  | 16:08 | <Ryan_Lane> | pooling ssl3002 | [production] | 
            
  | 16:06 | <Ryan_Lane> | installing new wikimedia-task-dns-auth package on dobson | [production] | 
            
  | 16:03 | <Ryan_Lane> | pooled ssl3001, depooling maerlant | [production] | 
            
  | 16:02 | <Ryan_Lane> | ipv6and4.labs moved to ssl3001 | [production] | 
            
  | 15:50 | <robh> | ssl3002-ssl3004 os installed | [production] | 
            
  | 15:33 | <Ryan_Lane> | restarting pybal on amslvs1 | [production] | 
            
  | 15:29 | <RoanKattouw> | Clearing /tmp directories on all mediawiki-installation hosts of files that haven't been accessed in more than 2 hours and are not in a subdirectory (this protects mw-cache files) | [production] | 
            
  | 15:26 | <robh> | ssl3002-ssl3004 installing | [production] | 
            
  | 15:26 | <robh> | cleaned up netboot.cfg (removing entries for servers that dont exist anymore), added in ssl esams range, updated dhcp | [production] | 
            
  | 15:25 | <Ryan_Lane> | restarting pybal on amslvs2 | [production] | 
            
  | 15:24 | <Ryan_Lane> | restarting pybal on amslvs4 | [production] | 
            
  | 15:22 | <robh> | ssl3001 installed with os, ready for puppet deployment | [production] | 
            
  | 15:18 | <RoanKattouw> | Using dsh to remove /tmp/mw-cache from all mediawiki-installation boxes. It's an obsolete caching dir with 300-400MB of dead data on each server | [production] | 
            
  | 15:14 | <Ryan_Lane> | restarting pybal on amslvs3 | [production] | 
            
  | 15:12 | <RoanKattouw> | Removed all /tmp files that hadn't been accessed in >48 hours on srv255 | [production] | 
            
  | 15:05 | <Ryan_Lane> | make that repository | [production] | 
            
  | 15:05 | <Ryan_Lane> | adding new wikimedia-task-dns-auth to repositoru | [production] | 
            
  | 15:03 | <py> | synchronized wmf-config/mc.php  'pushing out new mc.php to rotate out 2 more of a coming many boxes for upgrade to lucid' | [production] | 
            
  | 14:51 | <Ryan_Lane> | updating dns, adding mediawiki-lb using text's IP | [production] | 
            
  | 14:50 | <robh> | updating dns with ssl3001-ssl3004 | [production] | 
            
  | 14:32 | <py> | synchronized wmf-config/mc.php  'pushing out new mc.php to rotate out 2 of a coming many boxes for upgrade to lucid' | [production] | 
            
  | 13:55 | <mark> | Configured csw1-esams switchports for ssl3001-3006 | [production] | 
            
  | 12:37 | <mark> | shutdown server iris for decommissioning | [production] | 
            
  | 05:22 | <mutante> | sq69 -power up,dist-upgrade/kernel,reboot | [production] | 
            
  | 05:15 | <mutante> | sq67 -power up,dist-upgrade/kernel,reboot (varnish cache should be cleared on restart anyways) | [production] | 
            
  | 02:25 | <LocalisationUpdate> | completed (1.18) at Fri Sep 30 02:28:28 UTC 2011 | [production] | 
            
  | 02:23 | <LocalisationUpdate> | completed (1.17) at Fri Sep 30 02:25:42 UTC 2011 | [production] | 
            
  
    | 2011-09-29
      
      § | 
    
  | 22:22 | <maplebed> | nagios is busted; rerunning puppet to fix nagios cfg permissions | [production] | 
            
  | 21:12 | <preilly> | pushed update to 1.18 mobile frontend | [production] | 
            
  | 21:12 | <preilly> | synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php  'tuesday update mobile of frontend code part two' | [production] | 
            
  | 21:10 | <preilly> | push mobile frontend extension update | [production] | 
            
  | 21:10 | <preilly> | synchronized php/extensions/MobileFrontend/MobileFrontend.php  'tuesday update mobile of frontend code part two' | [production] | 
            
  | 21:02 | <preilly> | pushing this weeks changes for mobile frontend | [production] | 
            
  | 21:01 | <preilly> | synchronized php/extensions/MobileFrontend/MobileFrontend.php  'tuesday update mobile of frontend code' | [production] | 
            
  | 20:38 | <reedy> | synchronized php-1.18/resources/mediawiki.special/mediawiki.special.css  '[[rev:98444|r98444]]' | [production] | 
            
  | 20:38 | <reedy> | synchronized php-1.18/languages/messages/MessagesEn.php  '[[rev:98444|r98444]]' | [production] | 
            
  | 20:36 | <reedy> | synchronized php-1.18/skins/common/commonPrint.css  '[[rev:98444|r98444]]' | [production] | 
            
  | 20:36 | <reedy> | synchronized php-1.18/skins/common/shared.css  '[[rev:98444|r98444]]' | [production] | 
            
  | 20:21 | <notpeter> | adding mw28-58 to all dsh node groups that they belong in | [production] | 
            
  | 16:30 | <robh> | starting OS installation on mw28-mw58 | [production] | 
            
  | 14:02 | <Reedy> | Why isn't enwikibooks on 1.18wmf1 per the deployment schedule? | [production] | 
            
  | 04:09 | <mutante> | sq47 is "dazed and confused" - most likely broken RAM -> RT#1597 | [production] | 
            
  | 03:09 | <mutante> | knsq12 - power up,dist-upgrade/kernel,squid clean,reboot... | [production] | 
            
  | 02:24 | <LocalisationUpdate> | completed (1.18) at Thu Sep 29 02:26:48 UTC 2011 | [production] | 
            
  | 02:20 | <LocalisationUpdate> | completed (1.17) at Thu Sep 29 02:23:25 UTC 2011 | [production] | 
            
  | 01:27 | <tstarling> | synchronized php-1.17/includes/specials/SpecialIpblocklist.php  'privacy patch' | [production] | 
            
  | 01:24 | <tstarling> | synchronized php-1.17/includes/specials/SpecialIpblocklist.php  'privacy patch' | [production] | 
            
  | 01:21 | <tstarling> | synchronized php-1.17/includes/specials/SpecialIpblocklist.php  'privacy patch' | [production] | 
            
  | 00:54 | <Tim> | disabled api action=login logging on locke since it was apparently causing packet loss | [production] |