| 
      
        2023-06-15
      
      ยง
     | 
  
    
  | 22:28 | 
  <jhancock@cumin2002> | 
  END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-worker1154'] | 
  [production] | 
            
  | 22:21 | 
  <pt1979@cumin2002> | 
  END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | 
  [production] | 
            
  | 22:21 | 
  <pt1979@cumin2002> | 
  END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for ssw1-a1-codfw - pt1979@cumin2002" | 
  [production] | 
            
  | 22:18 | 
  <pt1979@cumin2002> | 
  START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add management record for ssw1-a1-codfw - pt1979@cumin2002" | 
  [production] | 
            
  | 22:14 | 
  <eevans@cumin1001> | 
  START - Cookbook sre.hosts.reimage for host cassandra-dev2002.codfw.wmnet with OS bullseye | 
  [production] | 
            
  | 22:14 | 
  <pt1979@cumin2002> | 
  START - Cookbook sre.dns.netbox | 
  [production] | 
            
  | 22:14 | 
  <pt1979@cumin2002> | 
  START - Cookbook sre.network.provision for device ssw1-a1-codfw.mgmt.codfw.wmnet | 
  [production] | 
            
  | 22:14 | 
  <jhancock@cumin2002> | 
  START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-worker1154'] | 
  [production] | 
            
  | 22:12 | 
  <jhancock@cumin2002> | 
  END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-worker1153'] | 
  [production] | 
            
  | 22:01 | 
  <jhancock@cumin2002> | 
  START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-worker1153'] | 
  [production] | 
            
  | 21:59 | 
  <eevans@cumin1001> | 
  END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2001.codfw.wmnet with OS bullseye | 
  [production] | 
            
  | 21:50 | 
  <jhancock@cumin2002> | 
  END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-worker1152'] | 
  [production] | 
            
  | 21:40 | 
  <jhancock@cumin2002> | 
  START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-worker1152'] | 
  [production] | 
            
  | 21:40 | 
  <jhancock@cumin2002> | 
  END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-worker1151'] | 
  [production] | 
            
  | 21:30 | 
  <jhancock@cumin2002> | 
  START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-worker1151'] | 
  [production] | 
            
  | 21:26 | 
  <eevans@cumin1001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2001.codfw.wmnet with reason: host reimage | 
  [production] | 
            
  | 21:24 | 
  <eevans@cumin1001> | 
  START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2001.codfw.wmnet with reason: host reimage | 
  [production] | 
            
  | 21:21 | 
  <jhancock@cumin2002> | 
  END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-worker1150'] | 
  [production] | 
            
  | 21:21 | 
  <jhancock@cumin2002> | 
  START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-worker1150'] | 
  [production] | 
            
  | 21:19 | 
  <jhancock@cumin1001> | 
  END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-worker1150'] | 
  [production] | 
            
  | 21:19 | 
  <jhancock@cumin1001> | 
  START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-worker1150'] | 
  [production] | 
            
  | 21:17 | 
  <jhancock@cumin2002> | 
  END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-worker1150'] | 
  [production] | 
            
  | 21:17 | 
  <jhancock@cumin2002> | 
  START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-worker1150'] | 
  [production] | 
            
  | 21:14 | 
  <jhancock@cumin2002> | 
  END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['an-worker1150'] | 
  [production] | 
            
  | 21:14 | 
  <thcipriani> | 
  parse1002 having ssh connection problems during backport window | 
  [production] | 
            
  | 21:13 | 
  <jhancock@cumin2002> | 
  START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-worker1150'] | 
  [production] | 
            
  | 21:13 | 
  <thcipriani@deploy1002> | 
  Finished scap: Backport for [[gerrit:930545|Revert "Targets: Use align:'after' instead of actionGroups" (T339292)]], [[gerrit:930541|HelpCompletionTool wasn't added to extension.json (T338254)]] (duration: 16m 09s) | 
  [production] | 
            
  | 21:12 | 
  <jhancock@cumin2002> | 
  END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['an-worker1150'] | 
  [production] | 
            
  | 21:12 | 
  <jhancock@cumin2002> | 
  START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-worker1150'] | 
  [production] | 
            
  | 21:12 | 
  <jhancock@cumin2002> | 
  END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['an-worker1150'] | 
  [production] | 
            
  | 21:11 | 
  <jhancock@cumin2002> | 
  START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-worker1150'] | 
  [production] | 
            
  | 21:11 | 
  <jhancock@cumin2002> | 
  END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-worker1150'] | 
  [production] | 
            
  | 21:08 | 
  <eevans@cumin1001> | 
  START - Cookbook sre.hosts.reimage for host cassandra-dev2001.codfw.wmnet with OS bullseye | 
  [production] | 
            
  | 20:58 | 
  <thcipriani@deploy1002> | 
  thcipriani and matmarex: Backport for [[gerrit:930545|Revert "Targets: Use align:'after' instead of actionGroups" (T339292)]], [[gerrit:930541|HelpCompletionTool wasn't added to extension.json (T338254)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet | 
  [production] | 
            
  | 20:57 | 
  <thcipriani@deploy1002> | 
  Started scap: Backport for [[gerrit:930545|Revert "Targets: Use align:'after' instead of actionGroups" (T339292)]], [[gerrit:930541|HelpCompletionTool wasn't added to extension.json (T338254)]] | 
  [production] | 
            
  | 20:54 | 
  <thcipriani@deploy1002> | 
  Finished scap: Backport for [[gerrit:930682|[uzwiki] Add the 'patroller' usergroup (T338826)]] (duration: 15m 27s) | 
  [production] | 
            
  | 20:52 | 
  <jhancock@cumin2002> | 
  START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-worker1150'] | 
  [production] | 
            
  | 20:50 | 
  <jhancock@cumin2002> | 
  END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-worker1149'] | 
  [production] | 
            
  | 20:42 | 
  <jhancock@cumin2002> | 
  START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-worker1149'] | 
  [production] | 
            
  | 20:40 | 
  <thcipriani@deploy1002> | 
  superpes and thcipriani: Backport for [[gerrit:930682|[uzwiki] Add the 'patroller' usergroup (T338826)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet | 
  [production] | 
            
  | 20:38 | 
  <thcipriani@deploy1002> | 
  Started scap: Backport for [[gerrit:930682|[uzwiki] Add the 'patroller' usergroup (T338826)]] | 
  [production] | 
            
  | 20:32 | 
  <jhancock@cumin2002> | 
  END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['an-worker1149'] | 
  [production] | 
            
  | 20:30 | 
  <thcipriani@deploy1002> | 
  Finished scap: Backport for [[gerrit:930639|Remove GDI survey from RU and JA wikis. (T338926)]] (duration: 16m 30s) | 
  [production] | 
            
  | 20:27 | 
  <jhancock@cumin2002> | 
  START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-worker1149'] | 
  [production] | 
            
  | 20:15 | 
  <thcipriani@deploy1002> | 
  essexigyan and thcipriani: Backport for [[gerrit:930639|Remove GDI survey from RU and JA wikis. (T338926)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet | 
  [production] | 
            
  | 20:13 | 
  <thcipriani@deploy1002> | 
  Started scap: Backport for [[gerrit:930639|Remove GDI survey from RU and JA wikis. (T338926)]] | 
  [production] | 
            
  | 19:06 | 
  <ladsgroup@deploy1002> | 
  Finished scap: Backport for [[gerrit:930672|Enable blocked domain list in testwiki and fawiki (T337431)]] (duration: 17m 40s) | 
  [production] | 
            
  | 18:50 | 
  <ladsgroup@deploy1002> | 
  ladsgroup: Backport for [[gerrit:930672|Enable blocked domain list in testwiki and fawiki (T337431)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet | 
  [production] | 
            
  | 18:48 | 
  <ladsgroup@deploy1002> | 
  Started scap: Backport for [[gerrit:930672|Enable blocked domain list in testwiki and fawiki (T337431)]] | 
  [production] | 
            
  | 18:48 | 
  <ryankemper> | 
  [WDQS] `ryankemper@wdqs2012:~$ sudo pool` | 
  [production] |