| 2023-06-22
      
      ยง | 
    
  | 12:32 | <root@cumin2002> | END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:32 | <root@cumin2002> | START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:28 | <root@cumin2002> | END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:28 | <root@cumin2002> | START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:28 | <root@cumin2002> | END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:27 | <root@cumin2002> | START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:26 | <root@cumin2002> | END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:26 | <root@cumin2002> | START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:25 | <root@cumin2002> | END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:25 | <root@cumin2002> | START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:25 | <root@cumin2002> | END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:17 | <root@cumin2002> | START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:06 | <root@cumin2002> | END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:06 | <root@cumin2002> | START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:06 | <root@cumin2002> | END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:06 | <root@cumin2002> | START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:05 | <root@cumin2002> | END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:04 | <root@cumin2002> | START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:04 | <root@cumin2002> | END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:04 | <root@cumin2002> | START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:03 | <root@cumin2002> | END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 12:03 | <root@cumin2002> | START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 11:57 | <root@cumin2002> | END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 11:57 | <root@cumin2002> | START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 11:45 | <root@cumin2002> | END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 11:45 | <root@cumin2002> | START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 11:44 | <root@cumin2002> | END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 11:44 | <root@cumin2002> | START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 11:41 | <root@cumin2002> | END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 11:41 | <root@cumin2002> | START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 11:37 | <jmm@cumin2002> | END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 11:37 | <jmm@cumin2002> | START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet | [production] | 
            
  | 11:36 | <jmm@cumin2002> | END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti-test2002.codfw.wmnet | [production] | 
            
  | 11:35 | <jmm@cumin2002> | START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2002.codfw.wmnet | [production] | 
            
  | 11:33 | <jbond@cumin2002> | END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] | [production] | 
            
  | 11:33 | <jbond@cumin2002> | START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] | [production] | 
            
  | 11:32 | <jbond@cumin2002> | END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] | [production] | 
            
  | 11:32 | <jbond@cumin2002> | START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sessionstore2001.codfw.wmnet'] | [production] | 
            
  | 11:32 | <jbond@cumin2002> | END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['sessionstore2001'] | [production] | 
            
  | 11:32 | <jbond@cumin2002> | START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['sessionstore2001'] | [production] | 
            
  | 11:24 | <jmm@cumin2002> | END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of testvm2002.codfw.wmnet to plain | [production] | 
            
  | 11:23 | <jmm@cumin2002> | START - Cookbook sre.ganeti.changedisk for changing disk type of testvm2002.codfw.wmnet to plain | [production] | 
            
  | 10:42 | <isaranto@deploy1002> | helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . | [production] | 
            
  | 10:33 | <mvolz@deploy1002> | helmfile [eqiad] DONE helmfile.d/services/citoid: apply | [production] | 
            
  | 10:33 | <mvolz@deploy1002> | helmfile [eqiad] START helmfile.d/services/citoid: apply | [production] | 
            
  | 10:33 | <mvolz@deploy1002> | helmfile [codfw] DONE helmfile.d/services/citoid: apply | [production] | 
            
  | 10:32 | <mvolz@deploy1002> | helmfile [codfw] START helmfile.d/services/citoid: apply | [production] | 
            
  | 10:32 | <mvolz@deploy1002> | helmfile [staging] DONE helmfile.d/services/citoid: apply | [production] | 
            
  | 10:31 | <mvolz@deploy1002> | helmfile [staging] START helmfile.d/services/citoid: apply | [production] | 
            
  | 10:29 | <mvolz@deploy1002> | helmfile [staging] DONE helmfile.d/services/citoid: apply | [production] |