| 
      
        2016-05-13
      
      §
     | 
  
    
  | 23:54 | 
  <catrope@tin> | 
  Synchronized php-1.28.0-wmf.1/extensions/Echo/includes/api/ApiEchoNotifications.php: Do not reuse CentralAuth tokens (duration: 00m 25s) | 
  [production] | 
            
  | 22:54 | 
  <catrope@tin> | 
  Synchronized php-1.28.0-wmf.1/extensions/Echo/includes/api/ApiEchoNotifications.php: Logging live hack (duration: 00m 31s) | 
  [production] | 
            
  | 22:50 | 
  <catrope@tin> | 
  Synchronized php-1.28.0-wmf.1/extensions/Echo/includes/api/ApiEchoNotifications.php: More logging (duration: 00m 25s) | 
  [production] | 
            
  | 22:38 | 
  <catrope@tin> | 
  Synchronized php-1.28.0-wmf.1/extensions/Flow/includes/Notifications/Formatter.php: Fix fatal for old Flow notifications (duration: 00m 26s) | 
  [production] | 
            
  | 22:34 | 
  <catrope@tin> | 
  Synchronized php-1.28.0-wmf.1/extensions/Echo/: Log failures to fetch cross-wiki notifications (duration: 00m 41s) | 
  [production] | 
            
  | 21:55 | 
  <catrope@tin> | 
  Synchronized php-1.28.0-wmf.1/extensions/Echo/Echo.php: Bump cache version now that cache pollution is hopefully fixed (duration: 00m 25s) | 
  [production] | 
            
  | 21:52 | 
  <catrope@tin> | 
  Synchronized php-1.28.0-wmf.1/extensions/Echo/: Fixes for cross-wiki notifications deployment fallout (duration: 00m 38s) | 
  [production] | 
            
  | 21:17 | 
  <bblack> | 
  cache_misc varnish3 downgrade complete (except 3007 + 1045 - those remain depooled/downtimed/puppet-disabled/etc, do not touch) | 
  [production] | 
            
  | 21:06 | 
  <catrope@tin> | 
  Synchronized wmf-config/CommonSettings.php: Enable $wgEchoCrossWikiNotifications on the right wikis (unused for now) (duration: 00m 28s) | 
  [production] | 
            
  | 20:09 | 
  <mattflaschen@tin> | 
  Synchronized wmf-config: Disable cross-wiki notifications entirely on non-SUL wikis and hide preference (duration: 00m 34s) | 
  [production] | 
            
  | 20:05 | 
  <bblack> | 
  starting varnish3 downgrade process for most of cache_misc | 
  [production] | 
            
  | 17:43 | 
  <godog> | 
  powercycle ms-be2008, sdb failed | 
  [production] | 
            
  | 15:41 | 
  <godog> | 
  bounce graphite-web on graphite1001, debugging carbon-c-relay stalls while sending metrics | 
  [production] | 
            
  | 13:57 | 
  <jynus> | 
  installing and testing sqlproxy on dbproxy1005 | 
  [production] | 
            
  | 13:44 | 
  <hashar> | 
  Jenkins zmq is all fine | 
  [production] | 
            
  | 13:43 | 
  <hashar> | 
  Restarted Jenkins, the zmq plugin daemon did not start | 
  [production] | 
            
  | 13:13 | 
  <ema> | 
  depooling cp3007 to try reproducing T134989 | 
  [production] | 
            
  | 11:28 | 
  <ema> | 
  repooling cp1061 | 
  [production] | 
            
  | 11:09 | 
  <moritzm> | 
  uploaded firejail 0.38-1+wmf1 for trusty-wikimedia to carbon | 
  [production] | 
            
  | 10:27 | 
  <ema> | 
  depooling cp1061 trying to reproduce T134989 | 
  [production] | 
            
  | 09:48 | 
  <godog> | 
  nodetool cleanup on restbase2006 T132976 | 
  [production] | 
            
  | 09:11 | 
  <godog> | 
  start backfilling cassandra metrics from graphite1001 to graphite1003 | 
  [production] | 
            
  | 05:27 | 
  <jynus> | 
  reimporting x1 on dbstore2002 | 
  [production] | 
            
  | 02:30 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Fri May 13 02:30:49 UTC 2016 (duration 8m 50s) | 
  [production] | 
            
  | 02:22 | 
  <mwdeploy@tin> | 
  scap sync-l10n completed (1.28.0-wmf.1) (duration: 08m 17s) | 
  [production] | 
            
  | 01:48 | 
  <awight|animal> | 
  rollback fundraising-tools from 53f6fe635dd8cc451b86788018e53f418b690b00 to e2978024e6f6b6881d087ac5d07e4c40f7374709 | 
  [production] | 
            
  | 00:53 | 
  <mutante> | 
  killing multiple icinga-wm processes on neon, running puppet. manually started and by puppet as well? | 
  [production] | 
            
  | 00:34 | 
  <demon@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: Remaining wikis to wmf.1 | 
  [production] | 
            
  | 00:17 | 
  <cwd> | 
  updated payments from b16c3cb0e12aacb49b91974d693c401dcaa9cca9 to 1af7208b4c7387a701f9ce83e33483d1e19213cb | 
  [production] | 
            
  | 00:13 | 
  <catrope@tin> | 
  Synchronized php-1.28.0-wmf.1/extensions/Echo/includes/api/ApiEchoNotifications.php: attempt to fix notices/fatals in Echo (duration: 00m 29s) | 
  [production] | 
            
  | 00:11 | 
  <catrope@tin> | 
  Synchronized php-1.27.0-wmf.23/extensions/Echo/includes/api/ApiEchoNotifications.php: attempt to fix notices/fatals in Echo (duration: 00m 25s) | 
  [production] | 
            
  
    | 
      
        2016-05-12
      
      §
     | 
  
    
  | 23:58 | 
  <chasemp> | 
  restart slapd instances.  we have some seemingly ldap issues in labs but it may not be the servers, could be bad merged changes. | 
  [production] | 
            
  | 23:54 | 
  <dereckson@tin> | 
  Synchronized php-1.28.0-wmf.1/extensions/CirrusSearch/includes/CirrusSearch.php: Adjust textcat data collection for AB test (T121542) (duration: 00m 26s) | 
  [production] | 
            
  | 23:43 | 
  <dereckson@tin> | 
  Synchronized wmf-config/CirrusSearch-common.php: Revert "A/B/C test of control vs textcat vs accept-lang + textcat" ([[Gerrit:288545]], 3/3) (duration: 00m 25s) | 
  [production] | 
            
  | 23:43 | 
  <dereckson@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Revert "A/B/C test of control vs textcat vs accept-lang + textcat" ([[Gerrit:288545]], 2/3) (duration: 00m 25s) | 
  [production] | 
            
  | 23:42 | 
  <dereckson@tin> | 
  Synchronized tests/cirrusTest.php: Revert "A/B/C test of control vs textcat vs accept-lang + textcat" ([[Gerrit:288545]], 1/3) (duration: 00m 24s) | 
  [production] | 
            
  | 23:33 | 
  <dereckson@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: A/B/C test of control vs textcat vs accept-lang + textcat ([[Gerrit:268048]]) (duration: 00m 25s) | 
  [production] | 
            
  | 23:23 | 
  <dereckson@tin> | 
  Synchronized wmf-config/CirrusSearch-common.php: A/B/C test of control vs textcat vs accept-lang + textcat ([[Gerrit:268048]]) (duration: 00m 25s) | 
  [production] | 
            
  | 23:22 | 
  <dereckson@tin> | 
  Synchronized tests/cirrusTest.php: A/B/C test of control vs textcat vs accept-lang + textcat (no-op) (duration: 00m 25s) | 
  [production] | 
            
  | 23:16 | 
  <dereckson@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Allow import from outreach to meta (T134788) (duration: 00m 27s) | 
  [production] | 
            
  | 23:10 | 
  <dereckson@tin> | 
  Synchronized wmf-config/throttle.php: UK EU edit-a-thon throttle rule ([[Gerrit:288377]], T134902) (duration: 00m 28s) | 
  [production] | 
            
  | 23:06 | 
  <dereckson@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Enable cross-wiki notifications by default in production ([[Gerrit:287035]], 2/2) (duration: 00m 29s) | 
  [production] | 
            
  | 23:06 | 
  <dereckson@tin> | 
  Synchronized wmf-config/InitialiseSettings-labs.php: Enable cross-wiki notifications by default in production ([[Gerrit:287035]], 1/2) (duration: 00m 42s) | 
  [production] | 
            
  | 22:17 | 
  <eileen> | 
  Updating civicrm from d32032965f2a45d24f1eaeb352eb6bfd6641753c to 6e3581692bb81c2c507612e7cd66dcdf78ef3cc0 | 
  [production] | 
            
  | 21:00 | 
  <aude@tin> | 
  Synchronized wmf-config/Wikibase.php: Update repoUrl setting for Wikibase (duration: 00m 28s) | 
  [production] | 
            
  | 21:00 | 
  <aude@tin> | 
  Synchronized wmf-config/Wikibase-labs.php: Update repoUrl setting for Wikibase (duration: 00m 32s) | 
  [production] |