| 
      
        2014-02-07
      
      §
     | 
  
    
  | 13:10 | 
  <paravoid> | 
  staggered restart of cp4xxx localssl, to deploy Ie94ccc (committed Oct 29th) | 
  [production] | 
            
  | 03:52 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at 2014-02-07 03:52:20+00:00 | 
  [production] | 
            
  | 03:50 | 
  <Ryan_Lane> | 
  removing myself from ops and wmf groups | 
  [production] | 
            
  | 03:12 | 
  <LocalisationUpdate> | 
  completed (1.23wmf13) at 2014-02-07 03:12:27+00:00 | 
  [production] | 
            
  | 02:27 | 
  <LocalisationUpdate> | 
  completed (1.23wmf12) at 2014-02-07 02:27:34+00:00 | 
  [production] | 
            
  | 01:39 | 
  <bd808> | 
  segfaults on mw1215 stopped after graceful restart | 
  [production] | 
            
  | 01:22 | 
  <ori> | 
  graceful'd mw1215 | 
  [production] | 
            
  | 01:19 | 
  <bd808> | 
  mw1215 logging 10-30 segfaults per minute since 19:56 | 
  [production] | 
            
  | 00:54 | 
  <kaldari> | 
  synchronized php-1.23wmf12/extensions/MobileFrontend  'syncing MobileFrontend make sure all the js is up to date' | 
  [production] | 
            
  | 00:49 | 
  <ori> | 
  graceful'd mw1142 | 
  [production] | 
            
  | 00:43 | 
  <krinkle> | 
  synchronized php-1.23wmf12/extensions/VisualEditor  'I1cc789596dd (re-sync, forgot to update inner submodule)' | 
  [production] | 
            
  | 00:31 | 
  <krinkle> | 
  synchronized php-1.23wmf12/extensions/VisualEditor  'I1cc789596dd' | 
  [production] | 
            
  | 00:29 | 
  <krinkle> | 
  synchronized php-1.23wmf13/extensions/VisualEditor  'I156b24551a40' | 
  [production] | 
            
  | 00:21 | 
  <ebernhardson> | 
  synchronized php-1.23wmf12/extensions/Flow/  'LD two patches to Flow - 1.23wmf12' | 
  [production] | 
            
  | 00:21 | 
  <ebernhardson> | 
  synchronized php-1.23wmf12/extensions/Echo/  'LD two patches to Echo - 1.23wmf12' | 
  [production] | 
            
  | 00:15 | 
  <ebernhardson> | 
  synchronized php-1.23wmf13/extensions/Flow/  'LD two patches to Flow-1.23wmf13' | 
  [production] | 
            
  | 00:15 | 
  <ebernhardson> | 
  synchronized php-1.23wmf13/extensions/Echo/  'LD two patches to Echo-1.23wmf13' | 
  [production] | 
            
  | 00:10 | 
  <maxsem> | 
  synchronized wmf-config/squid.php  'https://gerrit.wikimedia.org/r/111927' | 
  [production] | 
            
  
    | 
      
        2014-02-06
      
      §
     | 
  
    
  | 23:15 | 
  <mwalker> | 
  updating fundraising civicrm for thankyou messages from 309a69545b4c89b323bffc500f71bdee82c35e42 to 97a5146124168096148b6167e2968052b3dda468 | 
  [production] | 
            
  | 23:03 | 
  <Reedy> | 
  running sync-common on mw1142 | 
  [production] | 
            
  | 22:41 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  's1 db1055 resume groupLoadsByDB' | 
  [production] | 
            
  | 22:28 | 
  <demon> | 
  synchronized php-1.23wmf13/extensions/Math  'Unfataling wmf13 Math too' | 
  [production] | 
            
  | 22:28 | 
  <demon> | 
  updated /a/common/php-1.23wmf13 to {{Gerrit|I5ba48ca51}}: Reverting Math to known-good 2b8534793fad9db18fcdb9621dc8d79ff36fdeb1 | 
  [production] | 
            
  | 22:16 | 
  <demon> | 
  synchronized php-1.23wmf12/extensions/Math  'New math causes cache stampedes but without fatals' | 
  [production] | 
            
  | 21:54 | 
  <demon> | 
  synchronized php-1.23wmf13/extensions/Math  'Revert to known-good 2b85347 from wmf11' | 
  [production] | 
            
  | 21:52 | 
  <reedy> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedia back to 1.23wmf12, group0 to 1.23wmf13, Math reverted to 1.23wmf11 state in both branches | 
  [production] | 
            
  | 21:51 | 
  <demon> | 
  updated /a/common/php-1.23wmf13 to {{Gerrit|I233eaf25c}}: Revert "Add sequence support for externallinks table" | 
  [production] | 
            
  | 21:40 | 
  <demon> | 
  updated /a/common/php-1.23wmf12 to {{Gerrit|I91e982773}}: Update Echo and Flow | 
  [production] | 
            
  | 21:39 | 
  <demon> | 
  synchronized php-1.23wmf12/extensions/Math/  'Revert to known-good 2b85347 from wmf11' | 
  [production] | 
            
  | 21:17 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  's2 repool db1009 warm up' | 
  [production] | 
            
  | 19:46 | 
  <reedy> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: Revert earlier version changes | 
  [production] | 
            
  | 19:31 | 
  <reedy> | 
  started scap: run 2, should be a noop | 
  [production] | 
            
  | 19:28 | 
  <reedy> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf13 | 
  [production] | 
            
  | 19:13 | 
  <reedy> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf12 | 
  [production] | 
            
  | 18:36 | 
  <demon> | 
  synchronized wmf-config/CirrusSearch-labs.php  'Enabled interwiki searches for beta -- no-op in prod' | 
  [production] | 
            
  | 17:45 | 
  <paravoid> | 
  depooling mw1165 | 
  [production] | 
            
  | 17:43 | 
  <Reedy> | 
  mw1165 is segfaulting a lot | 
  [production] | 
            
  | 17:18 | 
  <reedy> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf12 till window | 
  [production] | 
            
  | 17:17 | 
  <reedy> | 
  finished scap: testwiki to 1.23wmf13 and build l10n cache (duration: 69m 51s) | 
  [production] | 
            
  | 16:50 | 
  <reedy> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: | 
  [production] | 
            
  | 16:48 | 
  <Nemo_bis> | 
  #Wikipedia and #Wikimedia wikis have been partly down for about 10 min, now recovered/ing | 
  [production] | 
            
  | 16:41 | 
  <reedy> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: | 
  [production] | 
            
  | 16:07 | 
  <reedy> | 
  started scap: testwiki to 1.23wmf13 and build l10n cache | 
  [production] | 
            
  | 16:04 | 
  <reedy> | 
  started scap: testwiki to 1.23wmf13 and build l10n cache | 
  [production] | 
            
  | 16:00 | 
  <reedy> | 
  started scap: testwiki to 1.23wmf13 and build l10n cache | 
  [production] | 
            
  | 15:57 | 
  <hashar> | 
  restarted jenkins by mistake :-( | 
  [production] | 
            
  | 15:28 | 
  <reedy> | 
  updated /a/common to {{Gerrit|I1e5ea52ae}}: s2 increase db1024 load after warm up | 
  [production] | 
            
  | 13:13 | 
  <paravoid> | 
  US/Canada pacific states being served by ulsfo | 
  [production] | 
            
  | 12:43 | 
  <paravoid> | 
  pointing the rest of East Asia (except CN) to ulsfo | 
  [production] | 
            
  | 11:14 | 
  <hashar> | 
  jenkins: added label <tt>hasTox</tt> on integration-slave01.pmtpa.wmflabs. Will let us run tox based Jenkins jobs there. | 
  [production] |