| 2014-02-07
      
      § | 
    
  | 22:22 | <mwalker> | updating payments wiki from 5f042ece4fa9d19293415ed41536ef93392c44ab to bad46aad4b28435d10570aac8173a7f8dca8751d for minfraud... again | [production] | 
            
  | 21:46 | <catrope> | synchronized php-1.23wmf13/extensions/MobileFrontend/includes/MobileFrontend.hooks.php  'Use feature flag for Minerva Beta Feature' | [production] | 
            
  | 21:46 | <catrope> | synchronized php-1.23wmf13/extensions/MobileFrontend/MobileFrontend.php  'Feature flag for Minerva Beta Feature' | [production] | 
            
  | 21:34 | <mwalker> | updated paymentswiki from fac21c9d394f6d7c9da172cee18c805304c8fe1b to 5f042ece4fa9d19293415ed41536ef93392c44ab for Amazon Recurring and Minfraud health check | [production] | 
            
  | 19:27 | <mutante> | running puppet on db9,pc1,pc2 etc, fixes mysqld process monitoring | [production] | 
            
  | 19:04 | <mutante> | revoked grosley's puppet cert | [production] | 
            
  | 18:49 | <akosiaris> | reenable puppet on mchenry as well as the exim cronjob. | [production] | 
            
  | 18:27 | <akosiaris> | temporarily disabling puppet on mchenry and then disable collect_exim_stats_via_gmetric cron. Seems like mchenry has not ganglia at all | [production] | 
            
  | 13:10 | <paravoid> | staggered restart of cp4xxx localssl, to deploy Ie94ccc (committed Oct 29th) | [production] | 
            
  | 03:52 | <LocalisationUpdate> | ResourceLoader cache refresh completed at 2014-02-07 03:52:20+00:00 | [production] | 
            
  | 03:50 | <Ryan_Lane> | removing myself from ops and wmf groups | [production] | 
            
  | 03:12 | <LocalisationUpdate> | completed (1.23wmf13) at 2014-02-07 03:12:27+00:00 | [production] | 
            
  | 02:27 | <LocalisationUpdate> | completed (1.23wmf12) at 2014-02-07 02:27:34+00:00 | [production] | 
            
  | 01:39 | <bd808> | segfaults on mw1215 stopped after graceful restart | [production] | 
            
  | 01:22 | <ori> | graceful'd mw1215 | [production] | 
            
  | 01:19 | <bd808> | mw1215 logging 10-30 segfaults per minute since 19:56 | [production] | 
            
  | 00:54 | <kaldari> | synchronized php-1.23wmf12/extensions/MobileFrontend  'syncing MobileFrontend make sure all the js is up to date' | [production] | 
            
  | 00:49 | <ori> | graceful'd mw1142 | [production] | 
            
  | 00:43 | <krinkle> | synchronized php-1.23wmf12/extensions/VisualEditor  'I1cc789596dd (re-sync, forgot to update inner submodule)' | [production] | 
            
  | 00:31 | <krinkle> | synchronized php-1.23wmf12/extensions/VisualEditor  'I1cc789596dd' | [production] | 
            
  | 00:29 | <krinkle> | synchronized php-1.23wmf13/extensions/VisualEditor  'I156b24551a40' | [production] | 
            
  | 00:21 | <ebernhardson> | synchronized php-1.23wmf12/extensions/Flow/  'LD two patches to Flow - 1.23wmf12' | [production] | 
            
  | 00:21 | <ebernhardson> | synchronized php-1.23wmf12/extensions/Echo/  'LD two patches to Echo - 1.23wmf12' | [production] | 
            
  | 00:15 | <ebernhardson> | synchronized php-1.23wmf13/extensions/Flow/  'LD two patches to Flow-1.23wmf13' | [production] | 
            
  | 00:15 | <ebernhardson> | synchronized php-1.23wmf13/extensions/Echo/  'LD two patches to Echo-1.23wmf13' | [production] | 
            
  | 00:10 | <maxsem> | synchronized wmf-config/squid.php  'https://gerrit.wikimedia.org/r/111927' | [production] | 
            
  
    | 2014-02-06
      
      § | 
    
  | 23:15 | <mwalker> | updating fundraising civicrm for thankyou messages from 309a69545b4c89b323bffc500f71bdee82c35e42 to 97a5146124168096148b6167e2968052b3dda468 | [production] | 
            
  | 23:03 | <Reedy> | running sync-common on mw1142 | [production] | 
            
  | 22:41 | <springle> | synchronized wmf-config/db-eqiad.php  's1 db1055 resume groupLoadsByDB' | [production] | 
            
  | 22:28 | <demon> | synchronized php-1.23wmf13/extensions/Math  'Unfataling wmf13 Math too' | [production] | 
            
  | 22:28 | <demon> | updated /a/common/php-1.23wmf13 to {{Gerrit|I5ba48ca51}}: Reverting Math to known-good 2b8534793fad9db18fcdb9621dc8d79ff36fdeb1 | [production] | 
            
  | 22:16 | <demon> | synchronized php-1.23wmf12/extensions/Math  'New math causes cache stampedes but without fatals' | [production] | 
            
  | 21:54 | <demon> | synchronized php-1.23wmf13/extensions/Math  'Revert to known-good 2b85347 from wmf11' | [production] | 
            
  | 21:52 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedia back to 1.23wmf12, group0 to 1.23wmf13, Math reverted to 1.23wmf11 state in both branches | [production] | 
            
  | 21:51 | <demon> | updated /a/common/php-1.23wmf13 to {{Gerrit|I233eaf25c}}: Revert "Add sequence support for externallinks table" | [production] | 
            
  | 21:40 | <demon> | updated /a/common/php-1.23wmf12 to {{Gerrit|I91e982773}}: Update Echo and Flow | [production] | 
            
  | 21:39 | <demon> | synchronized php-1.23wmf12/extensions/Math/  'Revert to known-good 2b85347 from wmf11' | [production] | 
            
  | 21:17 | <springle> | synchronized wmf-config/db-eqiad.php  's2 repool db1009 warm up' | [production] | 
            
  | 19:46 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Revert earlier version changes | [production] | 
            
  | 19:31 | <reedy> | started scap: run 2, should be a noop | [production] | 
            
  | 19:28 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf13 | [production] | 
            
  | 19:13 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf12 | [production] | 
            
  | 18:36 | <demon> | synchronized wmf-config/CirrusSearch-labs.php  'Enabled interwiki searches for beta -- no-op in prod' | [production] | 
            
  | 17:45 | <paravoid> | depooling mw1165 | [production] | 
            
  | 17:43 | <Reedy> | mw1165 is segfaulting a lot | [production] | 
            
  | 17:18 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf12 till window | [production] | 
            
  | 17:17 | <reedy> | finished scap: testwiki to 1.23wmf13 and build l10n cache (duration: 69m 51s) | [production] | 
            
  | 16:50 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: | [production] | 
            
  | 16:48 | <Nemo_bis> | #Wikipedia and #Wikimedia wikis have been partly down for about 10 min, now recovered/ing | [production] | 
            
  | 16:41 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: | [production] |