| 2013-01-21
      
      § | 
    
  | 23:11 | <Tim> | on sockpuppet: fixed puppet checkout, switching branch from mikepatch1 to production, and then did fetch&&rebase for good measure | [production] | 
            
  | 22:52 | <Ryan_Lane> | adding virt9-11 entries in dns | [production] | 
            
  | 22:49 | <reedy> | synchronized wmf-config/CommonSettings.php | [production] | 
            
  | 20:22 | <mark> | Rerouted AS43821->AS14907 traffic | [production] | 
            
  | 19:41 | <mark> | Restarting knsq* upload frontends manually | [production] | 
            
  | 19:34 | <paravoid> | deploying squid config for upload's /monitoring/ | [production] | 
            
  | 19:14 | <mark> | Restarting amssq* upload frontends in a slow loop | [production] | 
            
  | 19:11 | <mark> | Restarted oversized frontend on amssq50 | [production] | 
            
  | 18:55 | <paravoid> | starting knsq19 backend squid | [production] | 
            
  | 18:50 | <mark> | Restarted knsq16 backend minus two disks | [production] | 
            
  | 18:40 | <mark> | Started amssq56 squid instances | [production] | 
            
  | 18:38 | <mark> | Started knsq16 minus one disk | [production] | 
            
  | 18:37 | <mark> | Started knsq18 minus one disk | [production] | 
            
  | 18:32 | <mark> | power cycled amssq56 | [production] | 
            
  | 18:28 | <mark> | Rebooting knsq24 | [production] | 
            
  | 18:05 | <mark> | Rerouted AS43821->AS14907 traffic | [production] | 
            
  | 17:53 | <paravoid> | stopping knsq19 backend squid | [production] | 
            
  | 17:46 | <paravoid> | restarting knsq19 backend squid | [production] | 
            
  | 17:35 | <paravoid> | restarting pybal on lvs1002 | [production] | 
            
  | 17:35 | <cmjohnson1> | db1038 swapping bad disk (slot 2) with new disk | [production] | 
            
  | 17:27 | <mark> | Rerouted AS14907->AS43821 traffic | [production] | 
            
  | 16:36 | <paravoid> | depooling, restarting and repooling ms-fe2/3/4 one by one | [production] | 
            
  | 16:30 | <paravoid> | repooling ms-fe1 | [production] | 
            
  | 16:29 | <mark> | Rerouted AS43821->AS14907 traffic | [production] | 
            
  | 15:48 | <reedy> | synchronized php-1.21wmf8/includes/EditPage.php | [production] | 
            
  | 15:45 | <reedy> | synchronized php-1.21wmf7/includes/EditPage.php | [production] | 
            
  | 15:32 | <paravoid> | depooling ms-fe1 for testing | [production] | 
            
  | 14:17 | <hashar> | gallium: manually installed pyflakes {{gerrit|44974}} | [production] | 
            
  | 11:10 | <apergos> | nagios was dead over the weekend (config broken), fixed in puppet and on spence, now back in action | [production] | 
            
  | 09:57 | <hashar> | relaying Ryan: he restarted ldap on virt0 (was hung after server restart). nscld was properly falling back to virt1000 but ldap was stuck there too.  DNS got restarted. | [production] | 
            
  | 09:52 | <hashar> | jenkins: jobs refresh completed. | [production] | 
            
  | 08:14 | <hashar> | jenkins: updating all Jenkins jobs based on d31c92e of integration/jenkins-job-builder-config.git | [production] | 
            
  | 05:51 | <Tim> | on marmontel: removed MW-specific packages php5-wmerrors, php-luasandbox, php-wikidiff2 | [production] | 
            
  | 02:50 | <LocalisationUpdate> | completed (1.21wmf8) at Mon Jan 21 02:50:20 UTC 2013 | [production] | 
            
  | 02:27 | <LocalisationUpdate> | completed (1.21wmf7) at Mon Jan 21 02:26:59 UTC 2013 | [production] |