| 2012-03-10
      
      § | 
    
  | 19:28 | <binasher> | set sync_binlog = 1 on all current masters and eqiad dbs | [production] | 
            
  | 19:22 | <binasher> | reslaved db1033 | [production] | 
            
  | 07:03 | <mutante> | ran puppet on db1022, another one that works fine manually but somehow did not by itself | [production] | 
            
  | 05:11 | <mutante> | doing more (cp*, db*, msbe-* ,mw*) by hand / for loop | [production] | 
            
  | 05:01 | <mutante> | starting nagios-nrpe-server on all via dsh (fail to restart on config change issue) | [production] | 
            
  | 02:16 | <LocalisationUpdate> | completed (1.19) at Sat Mar 10 02:16:57 UTC 2012 | [production] | 
            
  | 01:07 | <maplebed> | started swiftcleaner on owa1 looking for (and purging) bad objects | [production] | 
            
  | 01:06 | <maplebed> | rebalanced the swift rings to finish decreasing traffic sent to ms1 and ms2 | [production] | 
            
  | 00:18 | <Ryan_Lane> | powercycling ssl1003 | [production] | 
            
  | 00:18 | <Ryan_Lane> | powercycling ssl1001 | [production] | 
            
  
    | 2012-03-09
      
      § | 
    
  | 20:34 | <notpeter> | stopping search indexer on searchidx2 for fresh rsync to searchidx1001 | [production] | 
            
  | 19:58 | <preilly> | pushed change to remove description from landing page | [production] | 
            
  | 19:57 | <preilly> | synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php  'changes for zero' | [production] | 
            
  | 18:59 | <Ryan_Lane> | sending test.m.wikipedia.org to the same place as test.wikipedia.org via squid | [production] | 
            
  | 18:58 | <awjrichards> | synchronized wmf-config/InitialiseSettings.php  'Fixing wgMobileUrlTemplate settings for domains that do not have .m. domains configured' | [production] | 
            
  | 18:48 | <reedy> | synchronized php-1.19/extensions/WikiLove/modules/ext.wikiLove/ext.wikiLove.css  '[[rev:113497|r113497]]' | [production] | 
            
  | 18:40 | <awjrichards> | synchronized wmf-config/CommonSettings.php  'Changing the way in which wgMobileUrlTemplate is configurable by InitialiseSettings.php' | [production] | 
            
  | 18:39 | <awjrichards> | synchronized wmf-config/InitialiseSettings.php  'Disabling wgMobileUrlTemplate for testwiki - hopefully for real this time' | [production] | 
            
  | 18:34 | <awjrichards> | synchronized wmf-config/CommonSettings.php  'Making wgMobileUrlTemplate configurable by InitialiseSettings.php' | [production] | 
            
  | 18:34 | <awjrichards> | synchronized wmf-config/InitialiseSettings.php  'Disabling wgMobileUrlTemplate for testwiki' | [production] | 
            
  | 17:40 | <awjrichards> | synchronized php/extensions/MobileFrontend/templates/ApplicationTemplate.php  '[[rev:113489|r113489]]' | [production] | 
            
  | 17:32 | <maplebed> | set swift storage device weight on ms2 to 0 and pushed out rings | [production] | 
            
  | 15:52 | <apergos> | cleared up a little bit of space on root partition of snapshot2, but that's about it.  I hope we never have 3 versions of mw in test at the same time, the tmp caches will kill us | [production] | 
            
  | 15:52 | <mark> | Turned off vcc_err_unref on all varnish servers, so varnish doesn't complain when ACLs/probes/backends are unused | [production] | 
            
  | 15:44 | <Jeff_Green> | hume apt upgrades, puppetd --test, switch to mysql 5.1.53-fb3753-wm1 | [production] | 
            
  | 06:38 | <Ryan_Lane> | reloading autofs on all labs instances | [production] | 
            
  | 06:13 | <Tim> | running svn cleanup on extdist trunk | [production] | 
            
  | 04:18 | <Tim> | switched php and wmf-deployment symlinks over to php-1.19 instead of php-1.18 | [production] | 
            
  | 04:18 | <Tim> | restarted morebots | [production] | 
            
  | 00:57 | <pp-pdf2> | updated pyfribidi to 0.11.0 fixing https://github.com/pediapress/pyfribidi/issues/2 | [production] | 
            
  | 00:57 | <pp-pdf3> | updated pyfribidi to 0.11.0 fixing https://github.com/pediapress/pyfribidi/issues/2 | [production] | 
            
  | 00:57 | <pp-pdf1> | updated pyfribidi to 0.11.0 fixing https://github.com/pediapress/pyfribidi/issues/2 | [production] | 
            
  | 00:38 | <preilly> | synchronized wmf-config/InitialiseSettings.php  'add ZeroRatedMobileAccess extension to mywiki' | [production] | 
            
  | 00:32 | <preilly> | synchronized php-1.19/extensions/MobileFrontend/javascripts/beta_opensearch.js  'fixes to code push' | [production] | 
            
  | 00:32 | <preilly> | synchronized php-1.19/extensions/MobileFrontend/javascripts/beta_opensearch.min.js  'fixes to code push' | [production] | 
            
  | 00:27 | <preilly> | synchronized php-1.19/extensions/MobileFrontend/templates/ApplicationTemplate.php  'fixes to code push' | [production] | 
            
  | 00:27 | <preilly> | synchronized php-1.19/extensions/MobileFrontend/javascripts/opensearch.js  'fixes to code push' | [production] | 
            
  | 00:27 | <preilly> | synchronized php-1.19/extensions/MobileFrontend/javascripts/opensearch.min.js  'fixes to code push' | [production] | 
            
  | 00:01 | <RobH> | oxygen install done, booting successfully after multiple tests, now running puppet for initial config | [production] | 
            
  | 00:01 | <K4-713> | updated the paypal IPN listener on aluminium to [[rev:1450|r1450]] | [production] | 
            
  
    | 2012-03-08
      
      § | 
    
  | 23:57 | <awjrichards> | synchronized php-1.19/extensions/MobileFrontend/templates/ApplicationTemplate.php  '[[rev:113428|r113428]]' | [production] | 
            
  | 23:56 | <tstarling> | synchronized wmf-config/CommonSettings.php  'per-wiki memory limit configuration, with extra memory for zh* for converter tables' | [production] | 
            
  | 23:55 | <tstarling> | synchronized wmf-config/InitialiseSettings.php  'per-wiki memory limit configuration, with extra memory for zh* for converter tables' | [production] | 
            
  | 23:42 | <mutante> | rebooting ms-be5 | [production] | 
            
  | 23:37 | <awjrichards> | synchronizing Wikimedia installation... : Updating MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments | [production] | 
            
  | 23:24 | <binasher> | streaming hotbacking of db1017 to db1033 - no snapshots of enwiki in eqiad til db1033 is back | [production] | 
            
  | 23:19 | <Tim> | started changing the php symlink to 1.19 instead of 1.18, but then changed my mind and changed it back. | [production] | 
            
  | 23:16 | <tstarling> | synchronizing Wikimedia installation... : | [production] | 
            
  | 23:07 | <tstarling> | synchronized php-1.19/extensions/ExtensionDistributor/svn-invoker.conf | [production] | 
            
  | 23:01 | <asher> | synchronized wmf-config/db.php  'returning db24 to service' | [production] |