| 
      
        2016-04-04
      
      §
     | 
  
    
  | 09:59 | 
  <moritzm> | 
  installing pcre3 updates | 
  [production] | 
            
  | 09:57 | 
  <godog> | 
  start expanding raid0 on restbase2003 | 
  [production] | 
            
  | 09:54 | 
  <ema> | 
  nginx rolling restart for openssl upgrade: cp1046, cp1052, cp1068, cp1071, cp1099 | 
  [production] | 
            
  | 09:53 | 
  <elukey> | 
  de-pooled aqs1001.eqiad.wmnet as pre-step for nodejs upgrade | 
  [production] | 
            
  | 09:41 | 
  <godog> | 
  depool restbase2003 before raid expansion | 
  [production] | 
            
  | 09:37 | 
  <godog> | 
  repool restbase2004 | 
  [production] | 
            
  | 09:33 | 
  <godog> | 
  deploy restbase ba39d2bcd2f5 to restbase2004 before repooling | 
  [production] | 
            
  | 09:26 | 
  <volans> | 
  Re-enabling Puppet on cluster mysql and parsercache to deploy change 279596, T111654 | 
  [production] | 
            
  | 09:10 | 
  <volans> | 
  Disabling Puppet on cluster mysql and parsercache to merge and test change 279596 on db2040, T111654 | 
  [production] | 
            
  | 08:50 | 
  <moritzm> | 
  installing gnupg updates | 
  [production] | 
            
  | 06:54 | 
  <moritzm> | 
  installing apt bugfix updates | 
  [production] | 
            
  | 04:18 | 
  <TimStarling> | 
  iridium came back up, but mcelog reports high CPU temperature prior to the shutdown | 
  [production] | 
            
  | 04:13 | 
  <TimStarling> | 
  attempting to turn iridium back on via drac. "getraclog" says it powered itself off after resetting four times | 
  [production] | 
            
  | 03:15 | 
  <greg-g> | 
  Phabricator is down | 
  [production] | 
            
  | 02:33 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Mon Apr  4 02:33:39 UTC 2016 (duration 8m 31s) | 
  [production] | 
            
  | 02:25 | 
  <mwdeploy@tin> | 
  sync-l10n completed (1.27.0-wmf.19) (duration: 11m 45s) | 
  [production] | 
            
  | 00:49 | 
  <krinkle@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 29s) | 
  [production] | 
            
  | 00:30 | 
  <krinkle@tin> | 
  Synchronized w/static.php: (no message) (duration: 00m 35s) | 
  [production] | 
            
  | 00:21 | 
  <Krinkle> | 
  mwscript deleteEqualMessages.php --wiki ruwiki (T45917) | 
  [production] | 
            
  | 00:06 | 
  <Krinkle> | 
  mwscript deleteEqualMessages.php --wiki metawiki --delete | 
  [production] | 
            
  
    | 
      
        2016-04-03
      
      §
     | 
  
    
  | 22:37 | 
  <Krinkle> | 
  mwscript deleteEqualMessages.php --wiki eswiki --lang-code ca (T45917) | 
  [production] | 
            
  | 22:37 | 
  <Krinkle> | 
  mwscript deleteEqualMessages.php --wiki eswiki (T45917) | 
  [production] | 
            
  | 22:34 | 
  <Krinkle> | 
  mwscript deleteEqualMessages.php --wiki eswiki | 
  [production] | 
            
  | 19:16 | 
  <Krenair> | 
  mw1146 began to respond with 503s to all requests, tried restarting apache2/hhvm and shortly afterwards it started working again | 
  [production] | 
            
  | 13:26 | 
  <reedy@tin> | 
  Synchronized wmf-config/InitialiseSettings-labs.php: labs copy upload thing. noooooop for prod (duration: 00m 33s) | 
  [production] | 
            
  | 13:02 | 
  <reedy@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: labs copy upload thing. noooooop for prod (duration: 00m 28s) | 
  [production] | 
            
  | 11:37 | 
  <ori@tin> | 
  Synchronized wmf-config/extension-list-labs: I0d081186: Also add Newsletter to extension-list-labs (duration: 00m 27s) | 
  [production] | 
            
  | 11:34 | 
  <ori@tin> | 
  Synchronized wmf-config/InitialiseSettings-labs.php: I3ffe65b8: Load the Newsletter extension on the beta cluster (2/2) (duration: 00m 33s) | 
  [production] | 
            
  | 11:34 | 
  <ori@tin> | 
  Synchronized wmf-config/CommonSettings-labs.php: I3ffe65b8: Load the Newsletter extension on the beta cluster (1/2) (duration: 00m 34s) | 
  [production] | 
            
  | 03:14 | 
  <mwdeploy@tin> | 
  sync-l10n completed (1.27.0-wmf.19) (duration: 56m 43s) | 
  [production] | 
            
  
    | 
      
        2016-04-02
      
      §
     | 
  
    
  | 20:28 | 
  <hashar> | 
  Restarted Jenkins on gallium | 
  [production] | 
            
  | 19:16 | 
  <dereckson@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: 350K celebration logo for cs.wikipedia (T131605) (duration: 00m 29s) | 
  [production] | 
            
  | 19:14 | 
  <dereckson@tin> | 
  Synchronized static/images/project-logos/cswiki.png: 350K celebration logo for cs.wikipedia (duration: 00m 33s) | 
  [production] | 
            
  | 10:45 | 
  <reedy@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: wikitech back to .19 | 
  [production] | 
            
  | 10:01 | 
  <reedy@tin> | 
  Synchronized php-1.27.0-wmf.19/extensions/OATHAuth: Fix for 2FA testing (duration: 00m 30s) | 
  [production] | 
            
  | 07:22 | 
  <krinkle@tin> | 
  Synchronized php-1.27.0-wmf.19/extensions/NavigationTiming/modules/ext.navigationTiming.js: T131565 (duration: 00m 33s) | 
  [production] | 
            
  | 02:32 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Sat Apr  2 02:32:42 UTC 2016 (duration 8m 36s) | 
  [production] | 
            
  | 02:24 | 
  <mwdeploy@tin> | 
  sync-l10n completed (1.27.0-wmf.19) (duration: 10m 48s) | 
  [production] | 
            
  | 02:03 | 
  <l10nupdate@tin> | 
  LocalisationUpdate failed (1.27.0-wmf.18) at 2016-04-02 02:03:32+00:00 | 
  [production] | 
            
  | 01:04 | 
  <krinkle@tin> | 
  Synchronized php-1.27.0-wmf.19/includes/specials/SpecialRedirect.php: T131328 (duration: 00m 39s) | 
  [production] | 
            
  
    | 
      
        2016-04-01
      
      §
     | 
  
    
  | 23:43 | 
  <krinkle@tin> | 
  Synchronized php-1.27.0-wmf.19/extensions/MobileFrontend/includes/MobileFrontend.hooks.php: T131337 (duration: 00m 38s) | 
  [production] | 
            
  | 22:39 | 
  <Pchelolo> | 
  rolling restart restbase to pick up https://gerrit.wikimedia.org/r/#/c/280951/ config change | 
  [production] | 
            
  | 22:13 | 
  <mutante> | 
  hooft - shutdown | 
  [production] | 
            
  | 21:36 | 
  <mutante> | 
  hooft: stopping ganglia aggregators, remove from icinga/storedconfigclean | 
  [production] | 
            
  | 21:25 | 
  <mutante> | 
  hooft.esams - stop puppet, stop salt | 
  [production] | 
            
  | 20:57 | 
  <mutante> | 
  bast3001 - sign puppet certs, salt keys | 
  [production] | 
            
  | 19:20 | 
  <reedy@tin> | 
  Synchronized wmf-config/InitialiseSettings-labs.php: noop change for labs (duration: 00m 29s) | 
  [production] | 
            
  | 19:10 | 
  <mutante> | 
  booting slauerhoff into PXE, installing as new bast3001 | 
  [production] | 
            
  | 19:10 | 
  <reedy@tin> | 
  Synchronized wmf-config/CommonSettings.php: Replace old copyright image config (duration: 00m 32s) | 
  [production] | 
            
  | 19:09 | 
  <reedy@tin> | 
  Synchronized wmf-config/InitialiseSettings-labs.php: consistency (duration: 00m 28s) | 
  [production] |