| 
      
        2021-03-31
      
      §
     | 
  
    
  | 15:57 | 
  <arturo> | 
  rebooting `tools-mail-03` after enabling NFS (T267082, T278538) | 
  [tools] | 
            
  | 15:57 | 
  <arturo> | 
  rebooting `tools-mail-03` after enabling NFS (T | 
  [tools] | 
            
  | 15:04 | 
  <arturo> | 
  created MX record for `tools.wmcloud.org` pointing to `mail.tools.wmcloud.org` | 
  [tools] | 
            
  | 15:03 | 
  <arturo> | 
  created DNS A record `mail.tools.wmcloud.org` pointing to 185.15.56.63 | 
  [tools] | 
            
  | 14:56 | 
  <arturo> | 
  shutoff tools-mail-02 (T278538) | 
  [tools] | 
            
  | 14:54 | 
  <arturo> | 
  point floating IP 185.15.56.63 to tools-mail-03 (T278538) | 
  [tools] | 
            
  | 14:45 | 
  <arturo> | 
  created VM `tools-mail-03` as Debian Buster (T278538) | 
  [tools] | 
            
  | 14:39 | 
  <arturo> | 
  relocate some of the hiera keys for email server from project-level to prefix | 
  [tools] | 
            
  | 09:44 | 
  <dcaro> | 
  running disk performance test on etcd-4 (round2) | 
  [tools] | 
            
  | 09:05 | 
  <dcaro> | 
  running disk performance test on etcd-8 | 
  [tools] | 
            
  | 08:43 | 
  <dcaro> | 
  running disk performance test on etcd-4 | 
  [tools] | 
            
  
    | 
      
        2021-03-30
      
      §
     | 
  
    
  | 16:15 | 
  <bstorm> | 
  added `labstore::traffic_shaping::egress: 800mbps` to tools-static prefix T278539 | 
  [tools] | 
            
  | 15:44 | 
  <arturo> | 
  shutoff tools-static-12/13 (T278539) | 
  [tools] | 
            
  | 15:41 | 
  <arturo> | 
  point horizon web proxy `tools-static.wmflabs.org` to tools-static-14  (T278539) | 
  [tools] | 
            
  | 15:37 | 
  <arturo> | 
  add `mount_nfs: true` to tools-static prefix (T2778539) | 
  [tools] | 
            
  | 15:26 | 
  <arturo> | 
  create VM tools-static-14 with Debian Buster image (T278539) | 
  [tools] | 
            
  | 12:19 | 
  <arturo> | 
  introduce horizon proxy `deb-tools.wmcloud.org` (T278436) | 
  [tools] | 
            
  | 12:15 | 
  <arturo> | 
  shutdown tools-sgebastion-09 (stretch) | 
  [tools] | 
            
  | 11:05 | 
  <arturo> | 
  created VM `tools-sgebastion-10` as Debian Buster (T275865) | 
  [tools] | 
            
  | 11:04 | 
  <arturo> | 
  created server group `tools-bastion` with anti-affinity policy | 
  [tools] | 
            
  
    | 
      
        2021-03-25
      
      §
     | 
  
    
  | 19:30 | 
  <bstorm> | 
  forced deletion of all jobs stuck in a deleting state T277653 | 
  [tools] | 
            
  | 17:46 | 
  <arturo> | 
  rebooting tools-sgeexec-* nodes to account for new grid master (T277653) | 
  [tools] | 
            
  | 16:20 | 
  <arturo> | 
  rebuilding tools-sgegrid-master VM as debian buster (T277653) | 
  [tools] | 
            
  | 16:18 | 
  <arturo> | 
  icinga-downtime toolschecker for 2h | 
  [tools] | 
            
  | 16:05 | 
  <bstorm> | 
  failed over the tools grid to the shadow master T277653 | 
  [tools] | 
            
  | 13:36 | 
  <arturo> | 
  shutdown tools-sge-services-03 (T278354) | 
  [tools] | 
            
  | 13:33 | 
  <arturo> | 
  shutdown tools-sge-services-04 (T278354) | 
  [tools] | 
            
  | 13:31 | 
  <arturo> | 
  point aptly clients to `tools-services-05.tools.eqiad1.wikimedia.cloud` (hiera change) (T278354) | 
  [tools] | 
            
  | 12:58 | 
  <arturo> | 
  created VM `tools-services-05` as Debian Buster (T278354) | 
  [tools] | 
            
  | 12:51 | 
  <arturo> | 
  create cinder volume `tools-aptly-data` (T278354) | 
  [tools] |