| 2013-03-27
      
      § | 
    
  | 19:56 | <reedy> | Started syncing Wikimedia installation... : Rebuild localisation cache to fix he related ParserFunctions problems | [production] | 
            
  | 19:43 | <reedy> | synchronized php-1.21wmf12/extensions/ParserFunctions/ | [production] | 
            
  | 19:37 | <reedy> | Finished syncing Wikimedia installation... : Rebuild message cache for WikiData deployment | [production] | 
            
  | 19:05 | <reedy> | Started syncing Wikimedia installation... : Rebuild message cache for WikiData deployment | [production] | 
            
  | 18:50 | <cmjohnson1> | pushing mw1209-1220 into service | [production] | 
            
  | 18:42 | <reedy> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 18:39 | <reedy> | synchronized php-1.21wmf12/extensions/Wikibase | [production] | 
            
  | 18:27 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf12 | [production] | 
            
  | 18:23 | <mutante> | installing package upgrades on neon (Icinga), incl. ganglia-monitor,gmeta | [production] | 
            
  | 18:08 | <mutante> | bunch of mw* hosts: fix puppet runs | [production] | 
            
  | 18:08 | <kaldari> | synchronized php-1.21wmf12/extensions/Echo  'syncing Echo for wmf12' | [production] | 
            
  | 18:05 | <mutante> | mw1130 - kill and fix puppet runs | [production] | 
            
  | 17:41 | <Jeff_Green> | authdns-update, pdns on ns1 croaked, restarted | [production] | 
            
  | 15:05 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: be_x_oldwiki to 1.21wmf12 | [production] | 
            
  | 14:09 | <hashar> | jenkins has been restarted by mistake :-(  Job building will be unavailable for up to half an hour. | [production] | 
            
  | 13:58 | <hashar> | Jenkins: disabled the old Gerrit Trigger Plugin {{bug|46415}} | [production] | 
            
  | 09:56 | <Nikerabbit> | Testing Translate bug fixes on test.wikipedia.org | [production] | 
            
  | 05:20 | <Tim> | on fluorine: running mw-log-cleanup once to test Id2196e7b | [production] | 
            
  | 05:02 | <Ryan_Lane> | disabling Bayes-collector cron on sanger | [production] | 
            
  | 04:55 | <Ryan_Lane> | purging BayesLearning files older than 70 days on mchenry | [production] | 
            
  | 02:45 | <LocalisationUpdate> | completed (1.21wmf11) at Wed Mar 27 02:44:56 UTC 2013 | [production] | 
            
  | 02:19 | <LocalisationUpdate> | completed (1.21wmf12) at Wed Mar 27 02:19:11 UTC 2013 | [production] | 
            
  | 00:55 | <LeslieCarr> | ms-be3 was crashed with stack traces for swift-container | [production] | 
            
  | 00:54 | <LeslieCarr> | rebooting ms-be3 | [production] | 
            
  | 00:37 | <awight> | payments cluster updated from 4862fb8eef to f295d851c | [production] | 
            
  | 00:34 | <asher> | synchronized wmf-config/db-eqiad.php  'db1052 to full weight' | [production] | 
            
  | 00:31 | <asher> | synchronized wmf-config/db-eqiad.php  'pulling db1001, adding db1052 at a warmup weight' | [production] | 
            
  | 00:10 | <awight> | payments cluster updated from fa2e2 to a06a03f05 wheee! | [production] | 
            
  
    | 2013-03-26
      
      § | 
    
  | 23:54 | <awight> | payments cluster rolled back from e758102 to fa2e240b6e9 | [production] | 
            
  | 23:53 | <awight> | payments cluster updated from fa2e240b6e9 to e758102 | [production] | 
            
  | 23:53 | <Tim> | on all apaches: replacing /usr/local/apache/common-local/live-1.5 with a symlink to w | [production] | 
            
  | 23:44 | <tstarling> | synchronized w | [production] | 
            
  | 23:35 | <maxsem> | synchronized wmf-config  'https://gerrit.wikimedia.org/r/#/c/56019/' | [production] | 
            
  | 23:33 | <awight> | payment cluster rollback, from 4862fb8 to fa2e240b | [production] | 
            
  | 23:31 | <mutante> | restarting nrpe on professor | [production] | 
            
  | 23:28 | <olivneh> | synchronized php-1.21wmf12/extensions/NavigationTiming/modules/ext.navigationTiming.js  'Workaround for buggy NavTiming implementation in IE9 (bug 46474)' | [production] | 
            
  | 23:20 | <awight> | payments cluster updated from fa2e240b to 4862fb8 | [production] | 
            
  | 23:10 | <mutante> | db11 - disabling all notifications | [production] | 
            
  | 23:07 | <mutante> | mw13 - killing puppet and re-running | [production] | 
            
  | 23:05 | <reedy> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 23:00 | <mutante> | removing mysql packages from db1036 (broken dpkg, status "pi") | [production] | 
            
  | 22:55 | <mutante> | re-installing ganglia-monitor on mw1135, fix dpkg | [production] | 
            
  | 22:53 | <reedy> | synchronized docroot | [production] | 
            
  | 22:51 | <mutante> | starting udpprofile (carbon-cache.py) on professor | [production] | 
            
  | 22:50 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 22:49 | <mutante> | search21 - remove sun-j2sdk , replaced by oracle-jdsdk | [production] | 
            
  | 22:47 | <mutante> | search21 - broken DPKG: iF  sun-j2sdk1.6 | [production] | 
            
  | 22:45 | <mutante> | mw2 - broken DPKG: pi  librsvg2-2 , pi  librsvg2-bin | [production] | 
            
  | 22:39 | <LeslieCarr> | did a service apache2 start on mw108* | [production] | 
            
  | 22:28 | <reedy> | synchronized wmf-config/InitialiseSettings.php | [production] |