| 2012-05-25
      
      § | 
    
  | 15:18 | <beta-logmsgbot> | hashar: deleted jobrunner06 (precise), we just need one precise instance which will be jobrunner05 for now | [releng] | 
            
  | 14:45 | <beta-logmsgbot> | hashar: installing applicationserver::homeless and applicationserver::jobrunner on jobrunner06 | [releng] | 
            
  | 14:42 | <beta-logmsgbot> | hashar: installing applicationserver::homeless and applicationserver::jobrunner on jobrunner05 | [releng] | 
            
  | 14:27 | <beta-logmsgbot> | hashar: installed jobrunner05 and 06 using Ubuntu precise. Should let get a 0.27 ffmpeg installation for {{bug|37043}} | [releng] | 
            
  | 08:38 | <beta-logmsgbot> | root: on dbdump, deleted /etc/logrotate.d/mw-udp2log . Most probably in conflict with the one from deployment-feed which host the udp2log process | [releng] | 
            
  | 08:35 | <beta-logmsgbot> | root: gzipped /home/wikipedia/logs/archive/*20120525 see {{bug|37012}} :-( | [releng] | 
            
  | 08:23 | <beta-logmsgbot> | hashar: killed stuck jobs on jobrunner 02 and 03. Restarted loop. | [releng] | 
            
  
    | 2012-05-23
      
      § | 
    
  | 19:54 | <hashar> | rebooting apache20 following installation of imagescaler puppet class | [releng] | 
            
  | 19:49 | <hashar> | rebooting apache23 following installation of imagescaler puppet class | [releng] | 
            
  | 19:46 | <hashar> | rebooting apache22 following installation of imagescaler puppet class | [releng] | 
            
  | 19:15 | <hashar> | rebooting apache21 following installation of imagescaler puppet class | [releng] | 
            
  | 18:48 | <hashar> | Adding puppet class 'imagescaler' on all deployment-apacheXX instances in an attempt to fix thumbnails | [releng] | 
            
  | 18:36 | <labs-logs-bottie> | hashar: relocalisation cache done 367/367 languages rebuilt | [releng] | 
            
  | 18:28 | <labs-logs-bottie> | hashar: running `mwscript rebuildLocalisationCache.php --wiki=aawiki` for {{bug|36806}} | [releng] | 
            
  | 16:22 | <labs-logs-bottie> | hashar: delete all 3 webVideoTranscode jobs from enwiki database | [releng] | 
            
  | 16:10 | <hashar> | rebooted jobrunner03 to check everything works fine there | [releng] | 
            
  | 15:41 | <hashar> | deleting jobrunner01, it is crashed beyond repair. Will create a new one named jobrunner03 | [releng] | 
            
  | 15:27 | <hashar> | rebooting jobrunner01 to see how it goes | [releng] | 
            
  | 14:56 | <hashar> | stopped job runner on jobrunner01,  amounted /mnt/upload6 and /mnt/ | [releng] | 
            
  | 14:37 | <hashar> | running puppet on job runner to check change 8584 & 8585 worked | [releng] | 
            
  
    | 2012-05-22
      
      § | 
    
  | 13:49 | <hashar> | Deleting jobrunner03 and 04, not going to need them afterall | [releng] | 
            
  | 13:24 | <hashar> | deleting refreshLinks2 jobs from enwiki database | [releng] | 
            
  | 12:52 | <hashar> | deleting deployment-jobrunner{3,4} installation failed I got permission denied. Will recreate them using same hostname | [releng] | 
            
  | 11:55 | <hashar> | create two more job runner instances | [releng] | 
            
  | 10:09 | <hashar> | Remove deployment-webs instance which was meant to emulate the HTTPS access. Hacky and low priority for now, we will need to setup a nginx proxy one day to properly replicate the production infrastructure. | [releng] | 
            
  | 09:39 | <labs-logs-bottie> | hashar: rebooting jobrunner02 just to be sure it is properly loaded up | [releng] | 
            
  | 09:30 | <labs-logs-bottie> | hashar: jobrunner logs are available in /home/wikipedia/logs/runJobs.log now | [releng] | 
            
  | 09:25 | <hashar> | Fixed udp2log not able to add new log files in /home/wikipedia/log , that dir need to be writable by udp2log user!  See https://gerrit.wikimedia.org/r/8442 | https://bugzilla.wikimedia.org/37014 | [releng] | 
            
  | 08:54 | <hashar> | purged all logs from /home/wikipedia/logs/archive/ just to be safe | [releng] |