| 
      
        2016-03-16
      
      §
     | 
  
    
  | 09:04 | 
  <moritzm> | 
  rebooting stat1002/stat1003 for kernel upgrade | 
  [production] | 
            
  | 08:53 | 
  <_joe_> | 
  restarting hhvm on mw1007, stuck in a deadlock, apparently on HPHP::Treadmill::getAgeOldestRequest | 
  [production] | 
            
  | 08:53 | 
  <moritzm> | 
  rebooting labtest*2001 for kernel upgrade | 
  [production] | 
            
  | 08:45 | 
  <jynus@tin> | 
  Synchronized wmf-config/db-eqiad.php: Return s2 slaves to normal weight (duration: 00m 35s) | 
  [production] | 
            
  | 08:04 | 
  <moritzm> | 
  restarted mw1129/1002 (HHVM hung) | 
  [production] | 
            
  | 06:47 | 
  <ebernhardson> | 
  rebooting elastic2018.codfw.wmnet for kernel upgrade | 
  [production] | 
            
  | 06:10 | 
  <ebernhardson> | 
  rebooting elastic2017.codfw.wmnet for kernel upgrade | 
  [production] | 
            
  | 05:39 | 
  <ebernhardson> | 
  rebooting elastic2016.codfw.wmnet for kernel upgrade | 
  [production] | 
            
  | 04:42 | 
  <ebernhardson> | 
  rebooting elastic2015.codfw.wmnet for kernel upgrade | 
  [production] | 
            
  | 03:41 | 
  <ebernhardson> | 
  rebooting elastic2014.codfw.wmnet for kernel upgrade | 
  [production] | 
            
  | 03:19 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Wed Mar 16 03:19:26 UTC 2016 (duration 9m 41s) | 
  [production] | 
            
  | 03:09 | 
  <mwdeploy@tin> | 
  sync-l10n completed (1.27.0-wmf.17) (duration: 17m 36s) | 
  [production] | 
            
  | 02:45 | 
  <ebernhardson> | 
  rebooting elastic2013.codfw.wmnet for kernel upgrade | 
  [production] | 
            
  | 02:34 | 
  <mwdeploy@tin> | 
  sync-l10n completed (1.27.0-wmf.16) (duration: 13m 44s) | 
  [production] | 
            
  | 02:17 | 
  <ebernhardson> | 
  rebooting elastic2012.codfw.wmnet for kernel upgrade | 
  [production] | 
            
  | 01:32 | 
  <ebernhardson> | 
  rebooting elastic2011.codfw.wmnet for kernel upgrade | 
  [production] | 
            
  | 01:18 | 
  <maxsem@tin> | 
  Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: Debug (duration: 00m 26s) | 
  [production] | 
            
  | 00:57 | 
  <maxsem@tin> | 
  Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: Debug (duration: 00m 24s) | 
  [production] | 
            
  | 00:48 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.16/includes/jobqueue/utils/BacklinkJobUtils.php: I32ec0 (duration: 00m 30s) | 
  [production] | 
            
  | 00:46 | 
  <maxsem@tin> | 
  Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: Debug (duration: 00m 28s) | 
  [production] | 
            
  | 00:17 | 
  <maxsem@tin> | 
  Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: Log REQUEST_URI (duration: 00m 28s) | 
  [production] | 
            
  | 00:15 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.16/extensions/Wikidata/extensions/Wikibase/client/includes/Hooks/DataUpdateHookHandlers.php: Revert briefly disable Wikibase\Client\Hooks\DataUpdateHookHandlers::onParserCacheSaveComplete (duration: 00m 31s) | 
  [production] | 
            
  | 00:01 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.16/extensions/Wikidata/extensions/Wikibase/client/includes/Hooks/DataUpdateHookHandlers.php: Briefly disable Wikibase\Client\Hooks\DataUpdateHookHandlers::onParserCacheSaveComplete (duration: 00m 26s) | 
  [production] | 
            
  
    | 
      
        2016-03-15
      
      §
     | 
  
    
  | 23:43 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: (no message) (duration: 00m 24s) | 
  [production] | 
            
  | 23:40 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: AdHocDebug for JobQueueGroup->lazyPush() (exclude job runners) (duration: 00m 28s) | 
  [production] | 
            
  | 23:37 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: AdHocDebug for JobQueueGroup->lazyPush() (with ip) (duration: 00m 30s) | 
  [production] | 
            
  | 23:33 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.16/includes/jobqueue/JobQueueGroup.php: AdHocDebug for JobQueueGroup->lazyPush() (duration: 00m 28s) | 
  [production] | 
            
  | 23:12 | 
  <catrope@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Strip rather than hide HTML in MobileFrontend (duration: 00m 29s) | 
  [production] | 
            
  | 23:10 | 
  <catrope@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: WikidataPageBanner config changes (duration: 00m 33s) | 
  [production] | 
            
  | 22:40 | 
  <ori@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: I87c174cf83: Turn on RecursiveLinkPurge log bucket, for I29636c045 (duration: 00m 25s) | 
  [production] | 
            
  | 22:32 | 
  <krinkle@tin> | 
  Synchronized php-1.27.0-wmf.16/extensions/VisualEditor/extension.json: T129704 (duration: 00m 24s) | 
  [production] | 
            
  | 22:00 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.16/includes/api/ApiPurge.php: I29636c04: Add RecursiveLinkPurge log for API requests (duration: 00m 33s) | 
  [production] | 
            
  | 21:47 | 
  <twentyafterfour@tin> | 
  Purged l10n cache for 1.27.0-wmf.14 | 
  [production] | 
            
  | 21:44 | 
  <twentyafterfour@tin> | 
  rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.27.0-wmf.17 | 
  [production] | 
            
  | 21:24 | 
  <gehel> | 
  restarting elasticsearch server elastic2010.codfw.wmnet | 
  [production] | 
            
  | 20:48 | 
  <gehel> | 
  restarting elasticsearch server elastic2009.codfw.wmnet | 
  [production] | 
            
  | 20:45 | 
  <krinkle@tin> | 
  Synchronized php-1.27.0-wmf.16/extensions/MoodBar/MoodBar.php: T129978 (duration: 00m 53s) | 
  [production] | 
            
  | 20:36 | 
  <twentyafterfour> | 
  grr now it says .17, pebkac | 
  [production] | 
            
  | 20:35 | 
  <twentyafterfour> | 
  testwiki still shows 1.27.0-wmf.16 :( | 
  [production] | 
            
  | 20:35 | 
  <mutante> | 
  terbium - gzip nutcracker.log.1 for disk space | 
  [production] | 
            
  | 20:34 | 
  <twentyafterfour@tin> | 
  Finished scap: testwiki to 1.27.0-wmf.17 (duration: 52m 15s) | 
  [production] | 
            
  | 20:05 | 
  <papaul> | 
  labstore200[3-4] OS install on hold: making new partman recipe  | 
  [production] | 
            
  | 19:42 | 
  <twentyafterfour@tin> | 
  Started scap: testwiki to 1.27.0-wmf.17 | 
  [production] | 
            
  | 19:16 | 
  <gehel> | 
  restarting elasticsearch server elastic2008.codfw.wmnet | 
  [production] | 
            
  | 19:05 | 
  <akosiaris> | 
  enable puppet on neon | 
  [production] | 
            
  | 19:04 | 
  <_joe_> | 
  rolling restart of restbase in codfw | 
  [production] | 
            
  | 19:00 | 
  <mutante> | 
  alsafi back up with 4.4 kernel | 
  [production] | 
            
  | 18:57 | 
  <mutante> | 
  alsafi - url-downloader codfw - reboot | 
  [production] | 
            
  | 18:44 | 
  <gehel> | 
  restarting elasticsearch server elastic2007.codfw.wmnet | 
  [production] | 
            
  | 18:43 | 
  <akosiaris> | 
  disable puppet on neon for a few minutes while deploying https://gerrit.wikimedia.org/r/#/c/276199/7 | 
  [production] |