| 2014-02-07
      
      § | 
    
  | 03:50 | <Ryan_Lane> | removing myself from ops and wmf groups | [production] | 
            
  | 03:12 | <LocalisationUpdate> | completed (1.23wmf13) at 2014-02-07 03:12:27+00:00 | [production] | 
            
  | 02:27 | <LocalisationUpdate> | completed (1.23wmf12) at 2014-02-07 02:27:34+00:00 | [production] | 
            
  | 01:39 | <bd808> | segfaults on mw1215 stopped after graceful restart | [production] | 
            
  | 01:22 | <ori> | graceful'd mw1215 | [production] | 
            
  | 01:19 | <bd808> | mw1215 logging 10-30 segfaults per minute since 19:56 | [production] | 
            
  | 00:54 | <kaldari> | synchronized php-1.23wmf12/extensions/MobileFrontend  'syncing MobileFrontend make sure all the js is up to date' | [production] | 
            
  | 00:49 | <ori> | graceful'd mw1142 | [production] | 
            
  | 00:43 | <krinkle> | synchronized php-1.23wmf12/extensions/VisualEditor  'I1cc789596dd (re-sync, forgot to update inner submodule)' | [production] | 
            
  | 00:31 | <krinkle> | synchronized php-1.23wmf12/extensions/VisualEditor  'I1cc789596dd' | [production] | 
            
  | 00:29 | <krinkle> | synchronized php-1.23wmf13/extensions/VisualEditor  'I156b24551a40' | [production] | 
            
  | 00:21 | <ebernhardson> | synchronized php-1.23wmf12/extensions/Flow/  'LD two patches to Flow - 1.23wmf12' | [production] | 
            
  | 00:21 | <ebernhardson> | synchronized php-1.23wmf12/extensions/Echo/  'LD two patches to Echo - 1.23wmf12' | [production] | 
            
  | 00:15 | <ebernhardson> | synchronized php-1.23wmf13/extensions/Flow/  'LD two patches to Flow-1.23wmf13' | [production] | 
            
  | 00:15 | <ebernhardson> | synchronized php-1.23wmf13/extensions/Echo/  'LD two patches to Echo-1.23wmf13' | [production] | 
            
  | 00:10 | <maxsem> | synchronized wmf-config/squid.php  'https://gerrit.wikimedia.org/r/111927' | [production] | 
            
  
    | 2014-02-06
      
      § | 
    
  | 23:15 | <mwalker> | updating fundraising civicrm for thankyou messages from 309a69545b4c89b323bffc500f71bdee82c35e42 to 97a5146124168096148b6167e2968052b3dda468 | [production] | 
            
  | 23:03 | <Reedy> | running sync-common on mw1142 | [production] | 
            
  | 22:41 | <springle> | synchronized wmf-config/db-eqiad.php  's1 db1055 resume groupLoadsByDB' | [production] | 
            
  | 22:28 | <demon> | synchronized php-1.23wmf13/extensions/Math  'Unfataling wmf13 Math too' | [production] | 
            
  | 22:28 | <demon> | updated /a/common/php-1.23wmf13 to {{Gerrit|I5ba48ca51}}: Reverting Math to known-good 2b8534793fad9db18fcdb9621dc8d79ff36fdeb1 | [production] | 
            
  | 22:16 | <demon> | synchronized php-1.23wmf12/extensions/Math  'New math causes cache stampedes but without fatals' | [production] | 
            
  | 21:54 | <demon> | synchronized php-1.23wmf13/extensions/Math  'Revert to known-good 2b85347 from wmf11' | [production] | 
            
  | 21:52 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedia back to 1.23wmf12, group0 to 1.23wmf13, Math reverted to 1.23wmf11 state in both branches | [production] | 
            
  | 21:51 | <demon> | updated /a/common/php-1.23wmf13 to {{Gerrit|I233eaf25c}}: Revert "Add sequence support for externallinks table" | [production] | 
            
  | 21:40 | <demon> | updated /a/common/php-1.23wmf12 to {{Gerrit|I91e982773}}: Update Echo and Flow | [production] | 
            
  | 21:39 | <demon> | synchronized php-1.23wmf12/extensions/Math/  'Revert to known-good 2b85347 from wmf11' | [production] | 
            
  | 21:17 | <springle> | synchronized wmf-config/db-eqiad.php  's2 repool db1009 warm up' | [production] | 
            
  | 19:46 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Revert earlier version changes | [production] | 
            
  | 19:31 | <reedy> | started scap: run 2, should be a noop | [production] | 
            
  | 19:28 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf13 | [production] | 
            
  | 19:13 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf12 | [production] | 
            
  | 18:36 | <demon> | synchronized wmf-config/CirrusSearch-labs.php  'Enabled interwiki searches for beta -- no-op in prod' | [production] | 
            
  | 17:45 | <paravoid> | depooling mw1165 | [production] | 
            
  | 17:43 | <Reedy> | mw1165 is segfaulting a lot | [production] | 
            
  | 17:18 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf12 till window | [production] | 
            
  | 17:17 | <reedy> | finished scap: testwiki to 1.23wmf13 and build l10n cache (duration: 69m 51s) | [production] | 
            
  | 16:50 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: | [production] | 
            
  | 16:48 | <Nemo_bis> | #Wikipedia and #Wikimedia wikis have been partly down for about 10 min, now recovered/ing | [production] | 
            
  | 16:41 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: | [production] | 
            
  | 16:07 | <reedy> | started scap: testwiki to 1.23wmf13 and build l10n cache | [production] | 
            
  | 16:04 | <reedy> | started scap: testwiki to 1.23wmf13 and build l10n cache | [production] | 
            
  | 16:00 | <reedy> | started scap: testwiki to 1.23wmf13 and build l10n cache | [production] | 
            
  | 15:57 | <hashar> | restarted jenkins by mistake :-( | [production] | 
            
  | 15:28 | <reedy> | updated /a/common to {{Gerrit|I1e5ea52ae}}: s2 increase db1024 load after warm up | [production] | 
            
  | 13:13 | <paravoid> | US/Canada pacific states being served by ulsfo | [production] | 
            
  | 12:43 | <paravoid> | pointing the rest of East Asia (except CN) to ulsfo | [production] | 
            
  | 11:14 | <hashar> | jenkins: added label <tt>hasTox</tt> on integration-slave01.pmtpa.wmflabs. Will let us run tox based Jenkins jobs there. | [production] | 
            
  | 11:03 | <springle> | synchronized wmf-config/db-eqiad.php  's2 increase db1024 load after warm up' | [production] | 
            
  | 07:29 | <Ryan_Lane> | redeploying parsoid/deploy on wtp* | [production] |