| 
      
        2016-01-15
      
      §
     | 
  
    
  | 14:13 | 
  <mark> | 
  Temporarily paused md126 RAID check on labstore1001 (sync_action idle) | 
  [production] | 
            
  | 14:09 | 
  <chasemp> | 
  phab restart phd (reports as not running in phab itself) seems ok now | 
  [production] | 
            
  | 14:03 | 
  <mark> | 
  set sync_speed_min to 5000 for md126 on labstore1001 | 
  [production] | 
            
  | 13:28 | 
  <demon@tin> | 
  demon@tin Synchronized wmf-config/InitialiseSettings.php: w:he as import source for commonswiki (duration: 00m 49s) | 
  [production] | 
            
  | 12:17 | 
  <hashar> | 
  restarting Jenkins for plugins updates | 
  [production] | 
            
  | 11:07 | 
  <_joe_> | 
  re-enabled puppet on mw1013, restarted HHVM to make it pick up our latest changes | 
  [production] | 
            
  | 10:01 | 
  <moritzm> | 
  installed ganeti security updates | 
  [production] | 
            
  | 09:18 | 
  <moritzm> | 
  installed git security updates on all jessie systems | 
  [production] | 
            
  | 03:10 | 
  <l10nupdate@tin> | 
  l10nupdate@tin ResourceLoader cache refresh completed at Fri Jan 15 03:10:09 UTC 2016 (duration 6m 48s) | 
  [production] | 
            
  | 03:03 | 
  <mwdeploy@tin> | 
  mwdeploy@tin sync-l10n completed (1.27.0-wmf.10) (duration: 16m 02s) | 
  [production] | 
            
  | 02:30 | 
  <krenair@tin> | 
  krenair@tin Synchronized php-1.27.0-wmf.10/includes/api/ApiQueryRecentChanges.php: https://gerrit.wikimedia.org/r/264231 (duration: 00m 42s) | 
  [production] | 
            
  | 02:29 | 
  <mwdeploy@tin> | 
  mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 14m 00s) | 
  [production] | 
            
  | 02:23 | 
  <YuviPanda> | 
  pull annualreport git repo on bromine for Krenair | 
  [production] | 
            
  | 01:00 | 
  <krenair@tin> | 
  krenair@tin Synchronized php-1.27.0-wmf.10/includes/api/ApiQueryWatchlist.php: https://gerrit.wikimedia.org/r/#/c/264224/ (duration: 00m 31s) | 
  [production] | 
            
  | 00:27 | 
  <krenair@tin> | 
  krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/263905/ (duration: 00m 32s) | 
  [production] | 
            
  | 00:24 | 
  <krenair@tin> | 
  krenair@tin Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 31s) | 
  [production] | 
            
  | 00:22 | 
  <krenair@tin> | 
  krenair@tin Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/264091/ (duration: 00m 32s) | 
  [production] | 
            
  | 00:06 | 
  <mobrovac> | 
  restbase started a dump of enwiki to populate storage with mobileapps renders | 
  [production] | 
            
  
    | 
      
        2016-01-14
      
      §
     | 
  
    
  | 23:56 | 
  <mobrovac> | 
  restbase end deploy of dac31a8c | 
  [production] | 
            
  | 23:49 | 
  <mobrovac> | 
  restbase start deploy of dac31a8c | 
  [production] | 
            
  | 22:17 | 
  <csteipp> | 
  deployed patch for T122807 | 
  [production] | 
            
  | 19:55 | 
  <ottomata> | 
  restarted eventlogging_sync script to insert batches of 1000  | 
  [production] | 
            
  | 19:31 | 
  <dduvall@tin> | 
  dduvall@tin rebuilt wikiversions.php and synchronized wikiversions files: rollback labswiki to wmf.9 | 
  [production] | 
            
  | 19:02 | 
  <dduvall@tin> | 
  dduvall@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.10 | 
  [production] | 
            
  | 18:40 | 
  <bblack> | 
  removing old eqiad misc-web IP (DNS switched 50h ago (not 26 like above), TTLs are max 1h) | 
  [production] | 
            
  | 18:39 | 
  <bblack> | 
  removing old eqiad misc-web IP (DNS switched 26h ago, TTLs are max 1h) | 
  [production] | 
            
  | 18:01 | 
  <paravoid> | 
  turning up BGP with Zayo in eqiad | 
  [production] | 
            
  | 16:25 | 
  <demon@tin> | 
  demon@tin Synchronized wmf-config/throttle.php: (no message) (duration: 00m 49s) | 
  [production] | 
            
  | 15:48 | 
  <moritzm> | 
  installed DHCP security updates across the fleet | 
  [production] | 
            
  | 14:44 | 
  <_joe_> | 
  powercycling mw1013, console stuck | 
  [production] | 
            
  | 11:28 | 
  <godog> | 
  bounce uwsgi on labmon1001 | 
  [production] | 
            
  | 11:18 | 
  <godog> | 
  upgrade graphite-carbon / graphite-web on labmon1001 | 
  [production] | 
            
  | 10:38 | 
  <_joe_> | 
  restarting hhvm on odd-numbered jobrunners | 
  [production] | 
            
  | 10:29 | 
  <moritzm> | 
  installed DHCP security updates on carbon | 
  [production] | 
            
  | 04:28 | 
  <paravoid> | 
  powercycling mw1005/mw1011 | 
  [production] | 
            
  | 04:24 | 
  <paravoid> | 
  restart hhvm on odd-numbered appservers | 
  [production] | 
            
  | 02:30 | 
  <mwdeploy@tin> | 
  mwdeploy@tin sync-l10n completed (1.27.0-wmf.9) (duration: 12m 21s) | 
  [production] | 
            
  | 01:32 | 
  <Krenair> | 
  Wikitech rolled back to wmf.9 due to T123583 | 
  [production] | 
            
  | 01:27 | 
  <krenair@tin> | 
  krenair@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message) | 
  [production] | 
            
  | 01:06 | 
  <mutante> | 
  mw1009 - restarted hhvm | 
  [production] | 
            
  | 01:00 | 
  <krenair@tin> | 
  krenair@tin Synchronized php-1.27.0-wmf.10/extensions/VisualEditor/extension.json: https://gerrit.wikimedia.org/r/#/c/264031/ (duration: 01m 35s) | 
  [production] | 
            
  | 00:30 | 
  <krenair@tin> | 
  krenair@tin Synchronized php-1.27.0-wmf.10/extensions/CirrusSearch/includes: https://gerrit.wikimedia.org/r/#q,263991,n,z (duration: 06m 08s) | 
  [production] | 
            
  | 00:11 | 
  <krenair@tin> | 
  krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/263804/ (duration: 00m 31s) | 
  [production] | 
            
  | 00:09 | 
  <krenair@tin> | 
  krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/263804/ (duration: 00m 31s) | 
  [production] | 
            
  | 00:08 | 
  <krenair@tin> | 
  krenair@tin Synchronized php-1.27.0-wmf.10/extensions/Echo/modules/echo.variables.less: https://gerrit.wikimedia.org/r/#/c/263767/ (duration: 00m 45s) | 
  [production] |