| 
      
        2022-06-02
      
      §
     | 
  
    
  | 14:02 | 
  <joal> | 
  Start browser_general_daily on airflow | 
  [analytics] | 
            
  | 13:19 | 
  <joal> | 
  Drop and recreate wmf_raw.mediawiki_page table (field removal) | 
  [analytics] | 
            
  | 12:44 | 
  <joal> | 
  Remove wrongly formatted interlanguage data | 
  [analytics] | 
            
  | 12:36 | 
  <joal> | 
  Kill interlanguage-daily oozie job after successfull move to airflow | 
  [analytics] | 
            
  | 12:15 | 
  <joal> | 
  Deploy interlanguage fix to airflow | 
  [analytics] | 
            
  | 09:56 | 
  <joal> | 
  Relaunch sqoop after having deployed a corrective patch | 
  [analytics] | 
            
  | 09:46 | 
  <joal> | 
  Manually mark interlaguage historical tasks failed in airflow | 
  [analytics] | 
            
  | 08:54 | 
  <joal> | 
  Deploy airflow with spark3 jobs | 
  [analytics] | 
            
  | 08:47 | 
  <joal> | 
  Merging 2 airflow  spark3 jobs now that their refinery counterpart is dpeloyed | 
  [analytics] | 
            
  | 08:07 | 
  <joal> | 
  Deploy refinery onto HDFS | 
  [analytics] | 
            
  | 07:26 | 
  <joal> | 
  Deploy refinery using scap | 
  [analytics] | 
            
  
    | 
      
        2022-05-31
      
      §
     | 
  
    
  | 18:48 | 
  <ottomata> | 
  sudo -u hdfs hdfs dfsadmin -safemode leave on an-master1001 | 
  [analytics] | 
            
  | 18:12 | 
  <ottomata> | 
  sudo service hadoop-hdfs-namenode start on an-master1002 | 
  [analytics] | 
            
  | 18:10 | 
  <ottomata> | 
  sudo -u hdfs hdfs dfsadmin -safemode enter | 
  [analytics] | 
            
  | 17:47 | 
  <btullis> | 
  starting namenode services on am-master1001 | 
  [analytics] | 
            
  | 17:44 | 
  <btullis> | 
  restarting the datanodes on all five of the affected hadoop workers. | 
  [analytics] | 
            
  | 17:43 | 
  <btullis> | 
  restarting journalnode service on each of the five hadoop workers with journals. | 
  [analytics] | 
            
  | 17:41 | 
  <btullis> | 
  resizing each journalnode with resize2fs | 
  [analytics] | 
            
  | 17:38 | 
  <btullis> | 
  sudo lvresize -L+20G analytics1069-vg/journalnode | 
  [analytics] | 
            
  | 17:38 | 
  <btullis> | 
  increasing each of the hadoop journalnodes by 20 GB | 
  [analytics] | 
            
  | 17:33 | 
  <ottomata> | 
  stop journalnodes and datanodes on 5 hadoop journalnode hosts | 
  [analytics] | 
            
  | 17:30 | 
  <btullis> | 
  stopped the hdfs-namenode service on an-master100[1-2] | 
  [analytics] | 
            
  | 15:36 | 
  <milimetric> | 
  dropped razzi databases and deleted HDFS directories (in trash) | 
  [analytics] | 
            
  | 06:26 | 
  <elukey> | 
  `elukey@an-master1001:~$ sudo systemctl reset-failed hadoop-clean-fairscheduler-event-logs.service` | 
  [analytics] |