| 
      
        2014-08-06
      
      §
     | 
  
    
  | 21:33 | 
  <hashar> | 
  Jenkins: moved mediawiki-core-regression-hhvm-master to run on Trusty instance | 
  [production] | 
            
  | 20:26 | 
  <hashar> | 
  Jenkins: downgraded ansicolor plugin from 0.4 to 0.3.1   Some colors.js function emits ANSI codes to reset the color which are not properly understood | 
  [production] | 
            
  | 20:06 | 
  <hashar> | 
  I have broke Zuul/Jenkins :-] | 
  [production] | 
            
  | 18:53 | 
  <hashar> | 
  Jenkins slow startup is {{bug|69197}} | 
  [production] | 
            
  | 18:50 | 
  <hashar> | 
  restarting jenkins | 
  [production] | 
            
  | 18:49 | 
  <hashar> | 
  Stopping Jenkins.  Reverting upgrade of artifact deployer plugin | 
  [production] | 
            
  | 18:10 | 
  <mutante> | 
  puppet-catalog-compiler says to "wait while Jenkins is getting ready to work" | 
  [production] | 
            
  | 17:20 | 
  <hashar> | 
  Jenkins process jobs again, the UI will take a bunch of hours to load though due to some issue when initializing | 
  [production] | 
            
  | 17:14 | 
  <hashar> | 
  killed Jenkins  | 
  [production] | 
            
  | 17:12 | 
  <_joe_> | 
  stopped the jobrunner on mw1053, was running in fcgi mode unpuppetized and with a broken vhost. Fixed it, it started spawning exceptions. DO NOT enable puppet again | 
  [production] | 
            
  | 17:02 | 
  <^d> | 
  jenkins restarted, was stuck | 
  [production] | 
            
  | 15:52 | 
  <hashar> | 
  Restarted Zuul and Zuul-merger on gallium to tweak logging settings {{gerrit|152118}} | 
  [production] | 
            
  | 11:30 | 
  <hoo> | 
  Synchronized wmf-config/CommonSettings.php: Grant 'centralauth-rename' to 'steward' (duration: 00m 24s) | 
  [production] | 
            
  | 11:26 | 
  <demon> | 
  Synchronized wmf-config/abusefilter.php: (no message) (duration: 00m 19s) | 
  [production] | 
            
  | 10:10 | 
  <hashar> | 
  Jenkins web interface is back up  | 
  [production] | 
            
  | 09:54 | 
  <demon> | 
  Synchronized wmf-config/abusefilter.php: abuse filter settings for fawiki (duration: 00m 21s) | 
  [production] | 
            
  | 07:33 | 
  <hashar> | 
  restarting Jenkins. It apparently like to parse the whole history on reload, so aborting that. | 
  [production] | 
            
  | 07:13 | 
  <hashar> | 
  Upgrading Jenkins plugin and restarting. | 
  [production] | 
            
  | 07:04 | 
  <hashar> | 
  upgrading Jenkins to latest LTS | 
  [production] | 
            
  | 03:11 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Wed Aug  6 03:10:06 UTC 2014 (duration 10m 5s) | 
  [production] | 
            
  | 02:30 | 
  <LocalisationUpdate> | 
  completed (1.24wmf16) at 2014-08-06 02:29:00+00:00 | 
  [production] | 
            
  | 02:17 | 
  <LocalisationUpdate> | 
  completed (1.24wmf15) at 2014-08-06 02:16:38+00:00 | 
  [production] | 
            
  
    | 
      
        2014-08-04
      
      §
     | 
  
    
  | 22:58 | 
  <bblack> | 
  rebooting ms-be1012 | 
  [production] | 
            
  | 21:47 | 
  <ottomata> | 
  reenabling puppet on analytics1027 | 
  [production] | 
            
  | 21:47 | 
  <jgage> | 
  all kafka brokers upraded to 0.8.1.1 and data replicated: done | 
  [production] | 
            
  | 20:37 | 
  <ottomata> | 
  stopping puppet on analytics1027 to temporarily disable camus cron job | 
  [production] | 
            
  | 19:07 | 
  <ottomata> | 
  starting upgrade of kafka cluster | 
  [production] | 
            
  | 19:02 | 
  <maxsem> | 
  Synchronized php-1.24wmf16/includes/User.php: https://gerrit.wikimedia.org/r/#/c/151691/ (duration: 00m 06s) | 
  [production] | 
            
  | 18:57 | 
  <jgage> | 
  beginning kafka upgrade: disabling puppet on brokers | 
  [production] | 
            
  | 13:17 | 
  <apergos> | 
  stopped labs rsync job from dataset1001, mount of labstore1003 was borked, removed 90GB of stuff on /mnt/data (= /) filesystem, restarted nfsd on dataset1001, dumps back to going | 
  [production] | 
            
  | 03:12 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Mon Aug  4 03:11:03 UTC 2014 (duration 11m 2s) | 
  [production] | 
            
  | 02:29 | 
  <LocalisationUpdate> | 
  completed (1.24wmf16) at 2014-08-04 02:27:58+00:00 | 
  [production] | 
            
  | 02:17 | 
  <LocalisationUpdate> | 
  completed (1.24wmf15) at 2014-08-04 02:16:46+00:00 | 
  [production] |