| 2011-02-10
      
      § | 
    
  | 23:10 | <Ryan_Lane> | rebooting nova-compute1 and nova-compute2 | [production] | 
            
  | 22:36 | <mark> | bits.wikimeda.org on the backend is now only served by a special application server cluster (srv191, srv192), load balanced by Varnish | [production] | 
            
  | 22:06 | <Ryan_Lane> | upgrading qemu, kvm, qemu-kvm, qemu-common, purging kvm-pxe, installing libjpeg62 on nova-compute1 | [production] | 
            
  | 22:05 | <Ryan_Lane> | installing libjpeg62 on nova-compute2, as it is a requirement for qemu-common | [production] | 
            
  | 22:04 | <Ryan_Lane> | upgrading those packages on nova-compute2 | [production] | 
            
  | 22:03 | <Ryan_Lane> | upgrading qemu, kvm, qemu-kvm, qemu-common, purging kvm-pxe | [production] | 
            
  | 19:58 | <mark> | Depooled sq67 for testing | [production] | 
            
  | 19:45 | <mark> | Depooled srv192 as normal app server, converted to bits app server (in puppet) | [production] | 
            
  | 19:41 | <Ryan_Lane> | deploying a new squid config to all squids | [production] | 
            
  | 19:35 | <mark> | Depooled srv191 as normal app server, converted to bits app server (in puppet) | [production] | 
            
  | 19:24 | <Ryan_Lane> | restarting squid processes on sq34 | [production] | 
            
  | 19:09 | <Ryan_Lane> | restarting gmetad on spence | [production] | 
            
  | 19:07 | <Ryan_Lane> | made a new ganglia group for api squids, made sq59 and sq60 aggregators for text squids, made sq31 and sq32 aggregators for api squid | [production] | 
            
  | 19:07 | <Ryan_Lane> | moved sq34 and sq35 to api squids | [production] | 
            
  | 19:07 | <Ryan_Lane> | depooling sq34, sq35 | [production] | 
            
  | 18:40 | <Ryan_Lane> | adding sq34 and sq35 to api squids | [production] | 
            
  | 18:35 | <Ryan_Lane> | powering off sq32 - has bad disks | [production] | 
            
  | 18:26 | <Ryan_Lane> | powercycling sq32, it's dead, again | [production] | 
            
  | 18:13 | <Ryan_Lane> | powercycling sq32 | [production] | 
            
  | 16:41 | <Ryan_Lane> | cleaning up logs on ekrem, and restarting apache | [production] | 
            
  | 15:30 | <^demon> | made the 1.17.dblist available with the others on noc.wikimedia.org | [production] | 
            
  | 12:22 | <tstarling> | synchronized php-1.17/load.php | [production] | 
            
  | 12:20 | <tstarling> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'fix wgCacheDirectory' | [production] | 
            
  | 11:57 | <tstarling> | synchronized php-1.17/includes/resourceloader/ResourceLoaderFileModule.php  'profiling fix' | [production] | 
            
  | 11:53 | <tstarling> | synchronized php-1.17/includes/resourceloader/ResourceLoader.php | [production] | 
            
  | 11:53 | <tstarling> | synchronized php-1.17/extensions/Gadgets/Gadgets_body.php  'profiling fix' | [production] | 
            
  | 11:46 | <tstarling> | synchronized php-1.17/includes/resourceloader/ResourceLoader.php  'profiling fix?' | [production] | 
            
  | 11:41 | <tstarling> | synchronized php-1.17/includes/resourceloader/ResourceLoader.php  'profiling fix?' | [production] | 
            
  | 07:09 | <tstarling> | synchronized php-1.17/load.php | [production] | 
            
  | 07:03 | <root> | ran sync-common-all | [production] | 
            
  | 06:56 | <tstarling> | synchronized php-1.17/StartProfiler.php  'splitting out test2 with 1:1 sample rate for profiling testing' | [production] | 
            
  | 04:34 | <Ryan_Lane> | installing scsi-tools on nova-*.tesla | [production] | 
            
  | 04:29 | <Ryan_Lane> | adding 50GB of storage to each nova-*.tesla vm | [production] | 
            
  | 04:05 | <Ryan_Lane> | installing corosync on nova-*.tesla | [production] | 
            
  | 04:04 | <Ryan_Lane> | installing sheepdog on nova-*.tesla | [production] | 
            
  | 03:35 | <RobH> | mailing lists restarted | [production] | 
            
  | 03:27 | <RobH> | correction, halting mail LIST servers, not all mail servers. | [production] | 
            
  | 03:27 | <RobH> | halting mail servers to clean out an email from archives, will resume normal service shortly | [production] | 
            
  | 02:53 | <Tim> | restarting all apaches to get apc.ini change | [production] | 
            
  | 02:31 | <Tim> | deploying new version of php5-apc package to all apaches, to change a single line in /etc/php5/conf.d/apc.ini | [production] | 
            
  | 00:12 | <catrope> | synchronized php-1.17/StartProfiler.php  'Put 1.17 wikis in their own profiling category' | [production] | 
            
  | 00:10 | <catrope> | synchronized live-1.5/MWVersion.php  'Define ONE_SEVENTEEN for wikis running 1.17' | [production] |