| 2013-02-27
      
      § | 
    
  | 22:36 | <mutante> | installing package upgrades on bast1001 | [production] | 
            
  | 22:16 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 21:44 | <RobH> | rdb1001 & rdb1002 allocated, network setup, and ready for handoff, per rt4608 | [production] | 
            
  | 21:36 | <RobH> | authdns-update for rdb1001-1002 | [production] | 
            
  | 21:36 | <RobH> | authdns-update for sdb1001-1002 | [production] | 
            
  | 21:31 | <Tim> | running resetUserTokens.php on all wikis, in screen on hume, to fix user_token field corruption ([[bugzilla:41586|bug 41586]]) | [production] | 
            
  | 21:26 | <tstarling> | synchronized php-1.21wmf10/includes/User.php  'user_token fixes' | [production] | 
            
  | 20:05 | <K4-713> | updated payments cluster to fcbf9211c9836b | [production] | 
            
  | 19:04 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf10 | [production] | 
            
  | 18:55 | <LeslieCarr> | rebooting arsenic | [production] | 
            
  | 18:42 | <reedy> | Finished syncing Wikimedia installation... : Rebuild message cache for EducationProgram | [production] | 
            
  | 18:25 | <reedy> | Started syncing Wikimedia installation... : Rebuild message cache for EducationProgram | [production] | 
            
  | 17:57 | <reedy> | synchronized php-1.21wmf10/extensions/EducationProgram/ | [production] | 
            
  | 17:51 | <K4-713> | updated payments cluster to eab489d0889a | [production] | 
            
  | 17:13 | <mutante> | rebooting fenari for kernel upgrade | [production] | 
            
  | 08:33 | <apergos> | restarted lucene search on search1015 | [production] | 
            
  | 08:04 | <reedy> | synchronized live-1.5/static-1.21wmf4/ | [production] | 
            
  | 08:04 | <reedy> | synchronized docroot/bits/static-1.21wmf4/ | [production] | 
            
  | 07:58 | <reedy> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 07:05 | <Krinkle> | done adding 1.21wmf3, 1.21wmf4, 1.21wmf5 ([[bugzilla:44570|bug 44570]]) | [production] | 
            
  | 07:04 | <krinkle> | Finished syncing Wikimedia installation... : | [production] | 
            
  | 06:29 | <apergos> | snapshot1002 down for reinstallation | [production] | 
            
  | 06:27 | <krinkle> | Started syncing Wikimedia installation... : | [production] | 
            
  | 06:15 | <Krinkle> | adding 1.21wmf3, 1.21wmf4, 1.21wmf5 ([[bugzilla:44570|bug 44570]]) | [production] | 
            
  | 06:15 | <Krinkle> | added 1.21wmf1, 1.21wmf2 ([[bugzilla:44570|bug 44570]]) | [production] | 
            
  | 05:35 | <apergos> | restarted lucene search on search1016 | [production] | 
            
  | 04:57 | <krinkle> | Finished syncing Wikimedia installation... : | [production] | 
            
  | 04:37 | <krinkle> | Started syncing Wikimedia installation... : | [production] | 
            
  | 02:57 | <LocalisationUpdate> | completed (1.21wmf9) at Wed Feb 27 02:57:22 UTC 2013 | [production] | 
            
  | 02:31 | <LocalisationUpdate> | completed (1.21wmf10) at Wed Feb 27 02:31:06 UTC 2013 | [production] | 
            
  | 01:35 | <Krinkle> | Bringing back checkouts of MediaWiki 1.21wmf5 - 1.21wmf1 on fenari for [[bugzilla:44570|bug 44570]] | [production] | 
            
  | 01:35 | <Jeff_Green> | dist-upgrade and reboot grosley | [production] | 
            
  | 00:25 | <tstarling> | cleared profiling data | [production] | 
            
  
    | 2013-02-26
      
      § | 
    
  | 23:29 | <maxsem> | synchronized php-1.21wmf9/extensions/MobileFrontend/  'https://gerrit.wikimedia.org/r/51053 https://gerrit.wikimedia.org/r/51057' | [production] | 
            
  | 23:27 | <maxsem> | synchronized php-1.21wmf10/extensions/MobileFrontend/  'https://gerrit.wikimedia.org/r/51053 https://gerrit.wikimedia.org/r/51057' | [production] | 
            
  | 23:04 | <K4-713> | synchronized payments cluster to 596b854787132c730 | [production] | 
            
  | 22:35 | <mutante> | installing package upgrades on fenari | [production] | 
            
  | 22:31 | <maxsem> | Finished syncing Wikimedia installation... : Weekly mobile deployment | [production] | 
            
  | 22:31 | <LeslieCarr> | nagios is dead!  spence.wikimedia.org if you want to access it directly | [production] | 
            
  | 22:16 | <mutante> | gracefulling eqiad Apaches to push redirect for wikivoyage.eu | [production] | 
            
  | 22:15 | <dzahn> | gracefulled all apaches | [production] | 
            
  | 22:15 | <maxsem> | Started syncing Wikimedia installation... : Weekly mobile deployment | [production] | 
            
  | 20:58 | <RobH> | stat1 puppet runs fail due to dan account error, i am working on it, but i also want to run out and get something to eat, so leaving in broken state for 10 minutes until i return to work on it | [production] | 
            
  | 20:21 | <mutante> | DNS update - add wikiovyage.eu | [production] | 
            
  | 19:34 | <Jeff_Green> | package updates & reboots of payments hosts | [production] | 
            
  | 19:13 | <paravoid> | freezing all exim4 bounces to restore normal email delivery | [production] | 
            
  | 18:58 | <reedy> | synchronized wmf-config/CommonSettings.php | [production] | 
            
  | 17:54 | <paravoid> | restarting opendj on sanger | [production] | 
            
  | 17:45 | <AaronSchulz> | Copying math files into ceph | [production] | 
            
  | 15:14 | <mark> | suspended i-000001d7 and i-0000039e for sending out malicious emails | [production] |