| 2012-07-16
      
      § | 
    
  | 23:03 | <pp-pdf1> | restarted nserve | [production] | 
            
  | 23:02 | <pp-pdf1> | fixed nserve handling of filenames with whitespace | [production] | 
            
  | 22:46 | <pp-pdf1> | restart all services | [production] | 
            
  | 22:45 | <pp-pdf2> | restart all services | [production] | 
            
  | 22:45 | <pp-pdf3> | restart all services | [production] | 
            
  | 22:45 | <pp-pdf1> | update mwlib to 0.13.9 | [production] | 
            
  | 22:45 | <pp-pdf2> | update mwlib to 0.13.9 | [production] | 
            
  | 22:45 | <pp-pdf3> | update mwlib to 0.13.9 | [production] | 
            
  | 22:44 | <pp-pdf2> | update greenlet to 0.4.0 | [production] | 
            
  | 22:44 | <pp-pdf1> | update greenlet to 0.4.0 | [production] | 
            
  | 22:44 | <pp-pdf3> | update greenlet to 0.4.0 | [production] | 
            
  | 22:44 | <pp-pdf2> | upgrade qserve to 0.2.8 | [production] | 
            
  | 22:44 | <pp-pdf3> | upgrade qserve to 0.2.8 | [production] | 
            
  | 22:43 | <pp-pdf1> | upgrade qserve to 0.2.8 | [production] | 
            
  | 22:20 | <awjrichards> | synchronized wmf-config/CommonSettings.php  'Disabling epub generation from Collection extension on all wikis except for simple and test' | [production] | 
            
  | 22:19 | <awjrichards> | synchronized wmf-config/InitialiseSettings.php  'Making epub generation for Collection extension configurable' | [production] | 
            
  | 18:26 | <AaronSchulz> | Running copyFileBackend.php for commons (shards 8-b) | [production] | 
            
  | 18:02 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.20wmf7 | [production] | 
            
  | 17:08 | <LeslieCarr> | reactivating XO transit connections on cr1-sdtpa | [production] | 
            
  | 17:03 | <LeslieCarr> | draining cr2-eqiad to cr1-sdtpa link for moving of fiber | [production] | 
            
  | 15:56 | <mark> | lvs3 is back up, and idling | [production] | 
            
  | 15:56 | <mutante> | fixing pt.planet after missing locale has been added | [production] | 
            
  | 15:48 | <mutante> | installing package upgrades on singer (planet) | [production] | 
            
  | 15:43 | <cmjohnson1> | mw8 shutting down and taking offline  to run Dell's dset program | [production] | 
            
  | 15:38 | <mutante> | planet - svn up the config | [production] | 
            
  | 15:29 | <mark> | Reinstalling lvs3 with Ubuntu Precise | [production] | 
            
  | 15:17 | <mark> | lvs5 is back up and serving traffic | [production] | 
            
  | 14:49 | <mark> | Reinstalling lvs5 with Ubuntu Precise | [production] | 
            
  | 14:45 | <mark> | Stopped PyBal on lvs5 to failover traffic to lvs1 | [production] | 
            
  | 14:35 | <maplebed> | adjusted swift rings; set new object servers to 20, new container servers to 100 | [production] | 
            
  | 14:33 | <mark> | lvs6 is back up and serving traffic | [production] | 
            
  | 14:12 | <mark> | Reinstalling lvs6 with Ubuntu Precise | [production] | 
            
  | 14:02 | <mark> | Stopped PyBal on lvs6 to failover traffic to lvs2 | [production] | 
            
  | 13:58 | <mutante> | mw8 - alright, most likely just needs new DIMM per cmjohnson | [production] | 
            
  | 13:50 | <mutante> | mw8 PHP fatal errors, running out of memory | [production] | 
            
  | 12:56 | <mutante> | sync-apache, graceful-all to fix wikipedia.cz redirect | [production] | 
            
  | 05:32 | <apergos> | reboot to upgrade kernel etc on mw8 since it's been flapping anyways | [production] | 
            
  | 02:46 | <LocalisationUpdate> | completed (1.20wmf6) at Mon Jul 16 02:46:14 UTC 2012 | [production] | 
            
  | 02:23 | <LocalisationUpdate> | completed (1.20wmf7) at Mon Jul 16 02:23:04 UTC 2012 | [production] |