| 2013-04-11
      
      § | 
    
  | 13:14 | <reedy> | synchronized wmf-config/  'Disable ClickTracking :D' | [production] | 
            
  | 13:08 | <mark> | Inserted python-extras packages into the precise-wikimedia APT repository (backport by hashar) | [production] | 
            
  | 10:10 | <hashar> | Jenkins is back up | [production] | 
            
  | 09:23 | <hashar> | restarting Jenkins to get the git verbose mode :( | [production] | 
            
  | 08:44 | <hashar> | jenkins: stop git cloning mw/core over and over by reusing existing clone {{gerrit|58502}}. That should help with I/O starvation on gallium. | [production] | 
            
  | 08:40 | <Nikerabbit> | translation memory database rebuild finished | [production] | 
            
  | 08:17 | <Nikerabbit> | emptying Translate solr instance and starting rebuilds | [production] | 
            
  | 08:15 | <nikerabbit> | synchronized php-1.22wmf1/extensions/Translate/  'to master' | [production] | 
            
  | 07:11 | <hashar> | synchronized docroot/noc/conf/highlight.php  'Revert highlight.php recent modifications {{bug|47112}} {{gerrit|58664}}' | [production] | 
            
  | 07:01 | <hashar> | Put a live hack on fenari for {{bug|47112}} | [production] | 
            
  | 02:16 | <LocalisationUpdate> | completed (1.22wmf1) at Thu Apr 11 02:16:35 UTC 2013 | [production] | 
            
  | 01:10 | <Ryan_Lane> | disabling puppet on labstore3/4 - making sure gluster won't die without them and we're going to rebuild them | [production] | 
            
  
    | 2013-04-10
      
      § | 
    
  | 23:30 | <LeslieCarr> | deactivated XO ipv6 transit | [production] | 
            
  | 22:25 | <Ryan_Lane> | added live hack to OpenStackManager on wikitech, ensuring that DynamicSidebar support is only enabled for logged-in users | [production] | 
            
  | 22:23 | <aaron> | synchronized wmf-config/jobqueue-eqiad.php  'moved refreshLinks to redis.' | [production] | 
            
  | 22:20 | <^demon|sick> | restarting gerrit, was hung | [production] | 
            
  | 22:15 | <Ryan_Lane> | enable DynamicSidebar support in OpenStackManager | [production] | 
            
  | 22:14 | <Ryan_Lane> | temporarily disabling DynamicSidebar support in OpenStackManager | [production] | 
            
  | 21:59 | <notpeter> | gracefulling apache on stafford | [production] | 
            
  | 21:35 | <aaron> | synchronized wmf-config/jobqueue-eqiad.php  'Moved htmlCacheUpdate jobs to redis' | [production] | 
            
  | 21:18 | <ottomata> | added python-statsd to apt repo for hashar | [production] | 
            
  | 21:18 | <aaron> | synchronized wmf-config/jobqueue-eqiad.php | [production] | 
            
  | 21:06 | <ottomata> | added python-voluptuous to apt repo (for hashar) | [production] | 
            
  | 20:46 | <ottomata> | added python-jsonschema to wikimedia apt repo: ori-l :) | [production] | 
            
  | 20:00 | <aaron> | synchronized wmf-config/jobqueue-eqiad.php  'Moved async upload jobs to redis' | [production] | 
            
  | 19:12 | <notpeter> | running puppetd --enable on all nodes via salt | [production] | 
            
  | 19:09 | <mwalker> | eqiad frack is now using clustered memcache and has site local configuration for DB, ActiveMQ, MemCacheD shtuff | [production] | 
            
  | 18:23 | <aaron> | synchronized wmf-config/jobqueue-eqiad.php  'Enabled use of redis for null test jobs' | [production] | 
            
  | 18:06 | <aaron> | rebuilt wikiversions.cdb and synchronized wikiversions files: Switched all remaning wikis to 1.22wmf1 | [production] | 
            
  | 17:45 | <RobH> | authdns-update to rename db1012 to stat1002 | [production] | 
            
  | 17:28 | <aaron> | synchronized php-1.22wmf1/maintenance/copyJobQueue.php  'deployed 54f74111901ddebdc69f26847e3904140b3723d5' | [production] | 
            
  | 17:23 | <LeslieCarr> | restart unresponsive cp1037 | [production] | 
            
  | 17:14 | <RobH> | authdns-update | [production] | 
            
  | 15:47 | <mark> | Ran dist-upgrade (for varnish upgrade) on cp1021-1036 | [production] | 
            
  | 15:40 | <demon> | synchronized wmf-config/CommonSettings.php  'Nostalgia ext' | [production] | 
            
  | 15:40 | <demon> | synchronized wmf-config/extension-list  'Nostalgia ext' | [production] | 
            
  | 15:38 | <demon> | synchronized php-1.22wmf1/extensions/Nostalgia | [production] | 
            
  | 14:33 | <maxsem> | synchronized wmf-config/mobile.php | [production] | 
            
  | 14:18 | <maxsem> | synchronized wmf-config  'Enable $wgMFVaryResources on test' | [production] | 
            
  | 14:12 | <mark> | Inserted varnish 3.0.3plus-rc1-wm10 packages into the APT repository | [production] | 
            
  | 12:22 | <apergos> | removed outright puppet.log on neon, was > 7gb and no room left on device | [production] | 
            
  | 07:53 | <krinkle> | synchronized php-1.22wmf1/resources/startup.js  'Ia54dd738b3ce0995fa' | [production] | 
            
  | 05:57 | <Tim> | on stafford: swapdeath repeat narrowly averted via killall ruby && /etc/init.d/apache2 stop | [production] | 
            
  | 05:34 | <Tim> | on brewster: proxy down due to full root partition, again. Will fix it properly this time. | [production] | 
            
  | 05:27 | <olivneh> | synchronized wmf-config/InitialiseSettings.php  'Add 'Zero' & 'Zero talk' namespaces to metawiki (If6b3ce5e4)' | [production] | 
            
  | 04:52 | <Tim> | stafford went into swapdeath. I killed all ruby processes a couple of times, and then eventually stopped apache2 for a while, while it recovered | [production] | 
            
  | 02:28 | <LocalisationUpdate> | completed (1.21wmf12) at Wed Apr 10 02:28:06 UTC 2013 | [production] | 
            
  | 02:17 | <LocalisationUpdate> | completed (1.22wmf1) at Wed Apr 10 02:17:19 UTC 2013 | [production] | 
            
  | 00:21 | <mutante> | removing xenon and db1012 from icinga configs, running puppetstoredconfigclean.rb on them, restarting icinga | [production] | 
            
  | 00:07 | <maxsem> | synchronized php-1.21wmf12/extensions/MobileFrontend/includes/MobileFrontend.hooks.php  'https://gerrit.wikimedia.org/r/#/c/58436/' | [production] |