|
2023-10-04
ยง
|
| 20:46 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1067.eqiad.wmnet with OS bullseye |
[production] |
| 20:46 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1066.eqiad.wmnet with OS bullseye |
[production] |
| 20:46 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1065.eqiad.wmnet with OS bullseye |
[production] |
| 20:46 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1064.eqiad.wmnet with OS bullseye |
[production] |
| 20:45 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1063.eqiad.wmnet with OS bullseye |
[production] |
| 20:45 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1062.eqiad.wmnet with OS bullseye |
[production] |
| 20:45 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1065.eqiad.wmnet with OS bullseye |
[production] |
| 20:21 |
<eileen> |
re-enable process control (more better hopefully) config revision changed from 89231b1b to d66626f6 |
[production] |
| 19:50 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Depooling db2164 (T343198)', diff saved to https://phabricator.wikimedia.org/P52826 and previous config saved to /var/cache/conftool/dbconfig/20231004-195023-arnaudb.json |
[production] |
| 19:50 |
<arnaudb@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 19:50 |
<arnaudb@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 19:50 |
<arnaudb@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance |
[production] |
| 19:49 |
<arnaudb@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance |
[production] |
| 19:49 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2163 (T343198)', diff saved to https://phabricator.wikimedia.org/P52825 and previous config saved to /var/cache/conftool/dbconfig/20231004-194946-arnaudb.json |
[production] |
| 19:34 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P52824 and previous config saved to /var/cache/conftool/dbconfig/20231004-193439-arnaudb.json |
[production] |
| 19:33 |
<eileen> |
config revision changed from 89231b1b to d66626f6 |
[production] |
| 19:30 |
<eileen> |
civicrm upgraded from 169c3288 to 44800fc0 |
[production] |
| 19:29 |
<eileen> |
config revision changed from 4ae7bd71 to 89231b1b |
[production] |
| 19:19 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P52823 and previous config saved to /var/cache/conftool/dbconfig/20231004-191933-arnaudb.json |
[production] |
| 19:19 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.provision for host ganeti-test2004.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 19:12 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1067.eqiad.wmnet with OS bullseye |
[production] |
| 19:04 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2163 (T343198)', diff saved to https://phabricator.wikimedia.org/P52822 and previous config saved to /var/cache/conftool/dbconfig/20231004-190427-arnaudb.json |
[production] |
| 18:54 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1063.eqiad.wmnet with OS bullseye |
[production] |
| 18:53 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1064.eqiad.wmnet with OS bullseye |
[production] |
| 18:19 |
<jhuneidi@deploy2002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.41.0-wmf.29 refs T347080 |
[production] |
| 18:19 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1066.eqiad.wmnet with OS bullseye |
[production] |
| 18:19 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1062.eqiad.wmnet with OS bullseye |
[production] |
| 18:09 |
<jhuneidi@deploy2002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.41.0-wmf.29 refs T347080 |
[production] |
| 17:52 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1067.eqiad.wmnet with OS bullseye |
[production] |
| 17:47 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM testreduce1002.eqiad.wmnet |
[production] |
| 17:43 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1067.eqiad.wmnet with OS bullseye |
[production] |
| 17:43 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.reboot-vm for VM testreduce1002.eqiad.wmnet |
[production] |
| 17:33 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1066.eqiad.wmnet with OS bullseye |
[production] |
| 17:33 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1065.eqiad.wmnet with OS bullseye |
[production] |
| 17:33 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1064.eqiad.wmnet with OS bullseye |
[production] |
| 17:33 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1063.eqiad.wmnet with OS bullseye |
[production] |
| 17:33 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host cloudvirt1062.eqiad.wmnet with OS bullseye |
[production] |
| 17:32 |
<jclark@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudvirt1066'] |
[production] |
| 17:32 |
<jclark@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudvirt1065'] |
[production] |
| 17:32 |
<jclark@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudvirt1064'] |
[production] |
| 17:32 |
<jclark@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudvirt1063'] |
[production] |
| 17:31 |
<jclark@cumin1001> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudvirt1062'] |
[production] |
| 17:27 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1063.eqiad.wmnet with OS bullseye |
[production] |
| 17:27 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1064.eqiad.wmnet with OS bullseye |
[production] |
| 17:22 |
<dcausse@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/rdf-streaming-updater: apply |
[production] |
| 17:22 |
<dcausse@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/rdf-streaming-updater: apply |
[production] |
| 17:16 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1066.eqiad.wmnet with OS bullseye |
[production] |
| 17:03 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1065.eqiad.wmnet with OS bullseye |
[production] |
| 17:03 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudvirt1062.eqiad.wmnet with OS bullseye |
[production] |
| 16:59 |
<fabfur> |
merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/963326 (T347837). `purged` daemon will be restarted by puppet in esams in the next 30m |
[production] |