| 
      
        2013-01-21
      
      §
     | 
  
    
  | 19:11 | 
  <mark> | 
  Restarted oversized frontend on amssq50 | 
  [production] | 
            
  | 18:55 | 
  <paravoid> | 
  starting knsq19 backend squid | 
  [production] | 
            
  | 18:50 | 
  <mark> | 
  Restarted knsq16 backend minus two disks | 
  [production] | 
            
  | 18:40 | 
  <mark> | 
  Started amssq56 squid instances | 
  [production] | 
            
  | 18:38 | 
  <mark> | 
  Started knsq16 minus one disk | 
  [production] | 
            
  | 18:37 | 
  <mark> | 
  Started knsq18 minus one disk | 
  [production] | 
            
  | 18:32 | 
  <mark> | 
  power cycled amssq56 | 
  [production] | 
            
  | 18:28 | 
  <mark> | 
  Rebooting knsq24 | 
  [production] | 
            
  | 18:05 | 
  <mark> | 
  Rerouted AS43821->AS14907 traffic | 
  [production] | 
            
  | 17:53 | 
  <paravoid> | 
  stopping knsq19 backend squid | 
  [production] | 
            
  | 17:46 | 
  <paravoid> | 
  restarting knsq19 backend squid | 
  [production] | 
            
  | 17:35 | 
  <paravoid> | 
  restarting pybal on lvs1002 | 
  [production] | 
            
  | 17:35 | 
  <cmjohnson1> | 
  db1038 swapping bad disk (slot 2) with new disk | 
  [production] | 
            
  | 17:27 | 
  <mark> | 
  Rerouted AS14907->AS43821 traffic | 
  [production] | 
            
  | 16:36 | 
  <paravoid> | 
  depooling, restarting and repooling ms-fe2/3/4 one by one | 
  [production] | 
            
  | 16:30 | 
  <paravoid> | 
  repooling ms-fe1 | 
  [production] | 
            
  | 16:29 | 
  <mark> | 
  Rerouted AS43821->AS14907 traffic | 
  [production] | 
            
  | 15:48 | 
  <reedy> | 
  synchronized php-1.21wmf8/includes/EditPage.php | 
  [production] | 
            
  | 15:45 | 
  <reedy> | 
  synchronized php-1.21wmf7/includes/EditPage.php | 
  [production] | 
            
  | 15:32 | 
  <paravoid> | 
  depooling ms-fe1 for testing | 
  [production] | 
            
  | 14:17 | 
  <hashar> | 
  gallium: manually installed pyflakes {{gerrit|44974}} | 
  [production] | 
            
  | 11:10 | 
  <apergos> | 
  nagios was dead over the weekend (config broken), fixed in puppet and on spence, now back in action | 
  [production] | 
            
  | 09:57 | 
  <hashar> | 
  relaying Ryan: he restarted ldap on virt0 (was hung after server restart). nscld was properly falling back to virt1000 but ldap was stuck there too.  DNS got restarted. | 
  [production] | 
            
  | 09:52 | 
  <hashar> | 
  jenkins: jobs refresh completed. | 
  [production] | 
            
  | 08:14 | 
  <hashar> | 
  jenkins: updating all Jenkins jobs based on d31c92e of integration/jenkins-job-builder-config.git | 
  [production] | 
            
  | 05:51 | 
  <Tim> | 
  on marmontel: removed MW-specific packages php5-wmerrors, php-luasandbox, php-wikidiff2 | 
  [production] | 
            
  | 02:50 | 
  <LocalisationUpdate> | 
  completed (1.21wmf8) at Mon Jan 21 02:50:20 UTC 2013 | 
  [production] | 
            
  | 02:27 | 
  <LocalisationUpdate> | 
  completed (1.21wmf7) at Mon Jan 21 02:26:59 UTC 2013 | 
  [production] | 
            
  
    | 
      
        2013-01-19
      
      §
     | 
  
    
  | 22:24 | 
  <MaxSem> | 
  Created ProofredPage table on aswikisource | 
  [production] | 
            
  | 19:22 | 
  <pgehres> | 
  synchronized wmf-config/reporting-setup.php  'Switching S:FundraiserStatistics read slave back to db1025' | 
  [production] | 
            
  | 02:46 | 
  <LocalisationUpdate> | 
  completed (1.21wmf8) at Sat Jan 19 02:46:06 UTC 2013 | 
  [production] | 
            
  | 02:24 | 
  <LocalisationUpdate> | 
  completed (1.21wmf7) at Sat Jan 19 02:24:18 UTC 2013 | 
  [production] | 
            
  | 01:54 | 
  <binasher> | 
  testing completed, no longer sending live traffic to eqiad | 
  [production] | 
            
  | 01:53 | 
  <binasher> | 
  i was sending >40 reqs/sec from europe and asia to eqiad apaches for the last hour for real world non-english testing | 
  [production] | 
            
  | 00:27 | 
  <mutante> | 
  DNS update - add wikivoyager.org/.de | 
  [production] | 
            
  | 00:08 | 
  <RobH> | 
  blog is fixed, in that its the software equivalent of spinning plates | 
  [production] | 
            
  | 00:04 | 
  <RoanKattouw> | 
  Running git fsck on mediawiki/core.git as nobody on manganese | 
  [production] |