| 2013-04-10
      
      § | 
    
  | 21:18 | <aaron> | synchronized wmf-config/jobqueue-eqiad.php | [production] | 
            
  | 21:06 | <ottomata> | added python-voluptuous to apt repo (for hashar) | [production] | 
            
  | 20:46 | <ottomata> | added python-jsonschema to wikimedia apt repo: ori-l :) | [production] | 
            
  | 20:00 | <aaron> | synchronized wmf-config/jobqueue-eqiad.php  'Moved async upload jobs to redis' | [production] | 
            
  | 19:12 | <notpeter> | running puppetd --enable on all nodes via salt | [production] | 
            
  | 19:09 | <mwalker> | eqiad frack is now using clustered memcache and has site local configuration for DB, ActiveMQ, MemCacheD shtuff | [production] | 
            
  | 18:23 | <aaron> | synchronized wmf-config/jobqueue-eqiad.php  'Enabled use of redis for null test jobs' | [production] | 
            
  | 18:06 | <aaron> | rebuilt wikiversions.cdb and synchronized wikiversions files: Switched all remaning wikis to 1.22wmf1 | [production] | 
            
  | 17:45 | <RobH> | authdns-update to rename db1012 to stat1002 | [production] | 
            
  | 17:28 | <aaron> | synchronized php-1.22wmf1/maintenance/copyJobQueue.php  'deployed 54f74111901ddebdc69f26847e3904140b3723d5' | [production] | 
            
  | 17:23 | <LeslieCarr> | restart unresponsive cp1037 | [production] | 
            
  | 17:14 | <RobH> | authdns-update | [production] | 
            
  | 15:47 | <mark> | Ran dist-upgrade (for varnish upgrade) on cp1021-1036 | [production] | 
            
  | 15:40 | <demon> | synchronized wmf-config/CommonSettings.php  'Nostalgia ext' | [production] | 
            
  | 15:40 | <demon> | synchronized wmf-config/extension-list  'Nostalgia ext' | [production] | 
            
  | 15:38 | <demon> | synchronized php-1.22wmf1/extensions/Nostalgia | [production] | 
            
  | 14:33 | <maxsem> | synchronized wmf-config/mobile.php | [production] | 
            
  | 14:18 | <maxsem> | synchronized wmf-config  'Enable $wgMFVaryResources on test' | [production] | 
            
  | 14:12 | <mark> | Inserted varnish 3.0.3plus-rc1-wm10 packages into the APT repository | [production] | 
            
  | 12:22 | <apergos> | removed outright puppet.log on neon, was > 7gb and no room left on device | [production] | 
            
  | 07:53 | <krinkle> | synchronized php-1.22wmf1/resources/startup.js  'Ia54dd738b3ce0995fa' | [production] | 
            
  | 05:57 | <Tim> | on stafford: swapdeath repeat narrowly averted via killall ruby && /etc/init.d/apache2 stop | [production] | 
            
  | 05:34 | <Tim> | on brewster: proxy down due to full root partition, again. Will fix it properly this time. | [production] | 
            
  | 05:27 | <olivneh> | synchronized wmf-config/InitialiseSettings.php  'Add 'Zero' & 'Zero talk' namespaces to metawiki (If6b3ce5e4)' | [production] | 
            
  | 04:52 | <Tim> | stafford went into swapdeath. I killed all ruby processes a couple of times, and then eventually stopped apache2 for a while, while it recovered | [production] | 
            
  | 02:28 | <LocalisationUpdate> | completed (1.21wmf12) at Wed Apr 10 02:28:06 UTC 2013 | [production] | 
            
  | 02:17 | <LocalisationUpdate> | completed (1.22wmf1) at Wed Apr 10 02:17:19 UTC 2013 | [production] | 
            
  | 00:21 | <mutante> | removing xenon and db1012 from icinga configs, running puppetstoredconfigclean.rb on them, restarting icinga | [production] | 
            
  | 00:07 | <maxsem> | synchronized php-1.21wmf12/extensions/MobileFrontend/includes/MobileFrontend.hooks.php  'https://gerrit.wikimedia.org/r/#/c/58436/' | [production] | 
            
  | 00:06 | <maxsem> | synchronized php-1.22wmf1/extensions/MobileFrontend/includes/MobileFrontend.hooks.php  'https://gerrit.wikimedia.org/r/#/c/58436/' | [production] | 
            
  
    | 2013-04-09
      
      § | 
    
  | 23:29 | <tstarling> | synchronized php-1.22wmf1/extensions/Nostalgia | [production] | 
            
  | 23:21 | <notpeter> | running upgrades on db56 and rebooting | [production] | 
            
  | 23:19 | <notpeter> | running upgrades on db64 and rebooting | [production] | 
            
  | 22:55 | <maxsem> | synchronized php-1.21wmf12/extensions/MobileFrontend/ | [production] | 
            
  | 22:52 | <maxsem> | synchronized php-1.22wmf1/extensions/MobileFrontend/ | [production] | 
            
  | 22:36 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 22:33 | <LeslieCarr> | clearing a few varnish caches - fyi if db load is seen to increase | [production] | 
            
  | 22:16 | <maxsem> | Finished syncing Wikimedia installation... : Weekly mobile deployment | [production] | 
            
  | 21:45 | <maxsem> | Started syncing Wikimedia installation... : Weekly mobile deployment | [production] | 
            
  | 21:44 | <hashar> | Zuul is slowly dequeuing a large number of jobs. There is currently rougly 30 minutes lag between change submission and actually processing of the job.   No idea why but investigating. | [production] | 
            
  | 19:56 | <Ryan_Lane> | installed Nuke on wikitech | [production] | 
            
  | 18:11 | <K4-713> | updated payments anti-fraud rules | [production] | 
            
  | 13:07 | <hashar> | jenkins plugin upgrade is tracked in {{bug|47040}}. Luckily git plugin has been fixed just in time ;-D  Just need to migrate the config history. | [production] | 
            
  | 12:37 | <hashar> | jenkins plugins seems to have been upgraded overnight :( | [production] | 
            
  | 10:26 | <mark> | Deployed experimental varnish build (3.0.3plus~rc1-wm10) on dysprosium and cp1022 | [production] | 
            
  | 08:53 | <hashar> | Jenkins tests for mediawiki/core are broken ({{bug|47031}}) due to a faulty change that has been merged in master branch. Root cause is the gating job not testing the proper change {{bug| 46723}} :/ | [production] | 
            
  | 08:43 | <Nikerabbit> | Got rid of local changes and undeployed commit related to noc | [production] | 
            
  | 08:41 | <nikerabbit> | synchronized wmf-config/InitialiseSettings.php  'Narayam and WebFonts' | [production] | 
            
  | 05:41 | <Tim> | on mw1149: enabled apache access log | [production] | 
            
  | 02:31 | <LocalisationUpdate> | completed (1.21wmf12) at Tue Apr  9 02:30:59 UTC 2013 | [production] |