| 
      
        2014-02-08
      
      §
     | 
  
    
  | 21:00 | 
  <aaron> | 
  synchronized php-1.23wmf12/extensions/Math  'd96b29ca8b17f35e7068f0d3a16b5e2644e084f9' | 
  [production] | 
            
  | 20:41 | 
  <aaron> | 
  synchronized php-1.23wmf13/extensions/Math  '4844f52139593f4a324bf99b74d7abb91aac2e54' | 
  [production] | 
            
  | 09:31 | 
  <apergos> | 
  powercycling ms-be1001, 'soft lockup cpu stuck' on console, no login prompt | 
  [production] | 
            
  | 02:47 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at 2014-02-08 02:47:46+00:00 | 
  [production] | 
            
  | 02:27 | 
  <LocalisationUpdate> | 
  completed (1.23wmf13) at 2014-02-08 02:27:14+00:00 | 
  [production] | 
            
  | 02:13 | 
  <LocalisationUpdate> | 
  completed (1.23wmf12) at 2014-02-08 02:13:35+00:00 | 
  [production] | 
            
  | 00:57 | 
  <Nemo_bis> | 
  Job queue rather long (400k on en.wiki), OTRS reports of password resets not being delivered (bug 43936?), almost no jobs run today according to gdash | 
  [production] | 
            
  
    | 
      
        2014-02-07
      
      §
     | 
  
    
  | 22:22 | 
  <mwalker> | 
  updating payments wiki from 5f042ece4fa9d19293415ed41536ef93392c44ab to bad46aad4b28435d10570aac8173a7f8dca8751d for minfraud... again | 
  [production] | 
            
  | 21:46 | 
  <catrope> | 
  synchronized php-1.23wmf13/extensions/MobileFrontend/includes/MobileFrontend.hooks.php  'Use feature flag for Minerva Beta Feature' | 
  [production] | 
            
  | 21:46 | 
  <catrope> | 
  synchronized php-1.23wmf13/extensions/MobileFrontend/MobileFrontend.php  'Feature flag for Minerva Beta Feature' | 
  [production] | 
            
  | 21:34 | 
  <mwalker> | 
  updated paymentswiki from fac21c9d394f6d7c9da172cee18c805304c8fe1b to 5f042ece4fa9d19293415ed41536ef93392c44ab for Amazon Recurring and Minfraud health check | 
  [production] | 
            
  | 19:27 | 
  <mutante> | 
  running puppet on db9,pc1,pc2 etc, fixes mysqld process monitoring | 
  [production] | 
            
  | 19:04 | 
  <mutante> | 
  revoked grosley's puppet cert | 
  [production] | 
            
  | 18:49 | 
  <akosiaris> | 
  reenable puppet on mchenry as well as the exim cronjob. | 
  [production] | 
            
  | 18:27 | 
  <akosiaris> | 
  temporarily disabling puppet on mchenry and then disable collect_exim_stats_via_gmetric cron. Seems like mchenry has not ganglia at all  | 
  [production] | 
            
  | 13:10 | 
  <paravoid> | 
  staggered restart of cp4xxx localssl, to deploy Ie94ccc (committed Oct 29th) | 
  [production] | 
            
  | 03:52 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at 2014-02-07 03:52:20+00:00 | 
  [production] | 
            
  | 03:50 | 
  <Ryan_Lane> | 
  removing myself from ops and wmf groups | 
  [production] | 
            
  | 03:12 | 
  <LocalisationUpdate> | 
  completed (1.23wmf13) at 2014-02-07 03:12:27+00:00 | 
  [production] | 
            
  | 02:27 | 
  <LocalisationUpdate> | 
  completed (1.23wmf12) at 2014-02-07 02:27:34+00:00 | 
  [production] | 
            
  | 01:39 | 
  <bd808> | 
  segfaults on mw1215 stopped after graceful restart | 
  [production] | 
            
  | 01:22 | 
  <ori> | 
  graceful'd mw1215 | 
  [production] | 
            
  | 01:19 | 
  <bd808> | 
  mw1215 logging 10-30 segfaults per minute since 19:56 | 
  [production] | 
            
  | 00:54 | 
  <kaldari> | 
  synchronized php-1.23wmf12/extensions/MobileFrontend  'syncing MobileFrontend make sure all the js is up to date' | 
  [production] | 
            
  | 00:49 | 
  <ori> | 
  graceful'd mw1142 | 
  [production] | 
            
  | 00:43 | 
  <krinkle> | 
  synchronized php-1.23wmf12/extensions/VisualEditor  'I1cc789596dd (re-sync, forgot to update inner submodule)' | 
  [production] | 
            
  | 00:31 | 
  <krinkle> | 
  synchronized php-1.23wmf12/extensions/VisualEditor  'I1cc789596dd' | 
  [production] | 
            
  | 00:29 | 
  <krinkle> | 
  synchronized php-1.23wmf13/extensions/VisualEditor  'I156b24551a40' | 
  [production] | 
            
  | 00:21 | 
  <ebernhardson> | 
  synchronized php-1.23wmf12/extensions/Flow/  'LD two patches to Flow - 1.23wmf12' | 
  [production] | 
            
  | 00:21 | 
  <ebernhardson> | 
  synchronized php-1.23wmf12/extensions/Echo/  'LD two patches to Echo - 1.23wmf12' | 
  [production] | 
            
  | 00:15 | 
  <ebernhardson> | 
  synchronized php-1.23wmf13/extensions/Flow/  'LD two patches to Flow-1.23wmf13' | 
  [production] | 
            
  | 00:15 | 
  <ebernhardson> | 
  synchronized php-1.23wmf13/extensions/Echo/  'LD two patches to Echo-1.23wmf13' | 
  [production] | 
            
  | 00:10 | 
  <maxsem> | 
  synchronized wmf-config/squid.php  'https://gerrit.wikimedia.org/r/111927' | 
  [production] | 
            
  
    | 
      
        2014-02-06
      
      §
     | 
  
    
  | 23:15 | 
  <mwalker> | 
  updating fundraising civicrm for thankyou messages from 309a69545b4c89b323bffc500f71bdee82c35e42 to 97a5146124168096148b6167e2968052b3dda468 | 
  [production] | 
            
  | 23:03 | 
  <Reedy> | 
  running sync-common on mw1142 | 
  [production] | 
            
  | 22:41 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  's1 db1055 resume groupLoadsByDB' | 
  [production] | 
            
  | 22:28 | 
  <demon> | 
  synchronized php-1.23wmf13/extensions/Math  'Unfataling wmf13 Math too' | 
  [production] | 
            
  | 22:28 | 
  <demon> | 
  updated /a/common/php-1.23wmf13 to {{Gerrit|I5ba48ca51}}: Reverting Math to known-good 2b8534793fad9db18fcdb9621dc8d79ff36fdeb1 | 
  [production] | 
            
  | 22:16 | 
  <demon> | 
  synchronized php-1.23wmf12/extensions/Math  'New math causes cache stampedes but without fatals' | 
  [production] | 
            
  | 21:54 | 
  <demon> | 
  synchronized php-1.23wmf13/extensions/Math  'Revert to known-good 2b85347 from wmf11' | 
  [production] | 
            
  | 21:52 | 
  <reedy> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedia back to 1.23wmf12, group0 to 1.23wmf13, Math reverted to 1.23wmf11 state in both branches | 
  [production] | 
            
  | 21:51 | 
  <demon> | 
  updated /a/common/php-1.23wmf13 to {{Gerrit|I233eaf25c}}: Revert "Add sequence support for externallinks table" | 
  [production] | 
            
  | 21:40 | 
  <demon> | 
  updated /a/common/php-1.23wmf12 to {{Gerrit|I91e982773}}: Update Echo and Flow | 
  [production] | 
            
  | 21:39 | 
  <demon> | 
  synchronized php-1.23wmf12/extensions/Math/  'Revert to known-good 2b85347 from wmf11' | 
  [production] | 
            
  | 21:17 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  's2 repool db1009 warm up' | 
  [production] | 
            
  | 19:46 | 
  <reedy> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: Revert earlier version changes | 
  [production] |