|
2024-02-22
ยง
|
| 17:39 |
<hnowlan@cumin1002> |
START - Cookbook sre.hosts.reimage for host mw1458.eqiad.wmnet with OS bullseye |
[production] |
| 17:39 |
<btullis@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply |
[production] |
| 17:36 |
<btullis@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply |
[production] |
| 17:35 |
<cmooney@cumin1002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host testvm2002.codfw.wmnet with OS bullseye |
[production] |
| 17:26 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P57748 and previous config saved to /var/cache/conftool/dbconfig/20240222-172632-arnaudb.json |
[production] |
| 17:11 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P57747 and previous config saved to /var/cache/conftool/dbconfig/20240222-171125-arnaudb.json |
[production] |
| 17:05 |
<topranks> |
disabling IPv6 RAs for private1-a-codfw vlan on codfw core routers T355544 |
[production] |
| 16:58 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Remove legacy codfw vc switches from synced hiera data after netbox status change - cmooney@cumin1002 - T355544" |
[production] |
| 16:57 |
<cmooney@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Remove legacy codfw vc switches from synced hiera data after netbox status change - cmooney@cumin1002 - T355544" |
[production] |
| 16:56 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1226 (T357189)', diff saved to https://phabricator.wikimedia.org/P57746 and previous config saved to /var/cache/conftool/dbconfig/20240222-165619-arnaudb.json |
[production] |
| 16:56 |
<topranks> |
disabling link from asw-a-codfw vc to ssw1-a1-codfw and ssw1-a8-codfw T355544 |
[production] |
| 16:54 |
<dancy@deploy2002> |
Finished scap: testing T357402 again (duration: 08m 58s) |
[production] |
| 16:54 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db1226 (T357189)', diff saved to https://phabricator.wikimedia.org/P57745 and previous config saved to /var/cache/conftool/dbconfig/20240222-165401-arnaudb.json |
[production] |
| 16:53 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:53 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:53 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1216.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:53 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1216.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:53 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1214 (T357189)', diff saved to https://phabricator.wikimedia.org/P57744 and previous config saved to /var/cache/conftool/dbconfig/20240222-165312-arnaudb.json |
[production] |
| 16:45 |
<dancy@deploy2002> |
Started scap: testing T357402 again |
[production] |
| 16:43 |
<dancy@deploy2002> |
sync-world aborted: testing T357402 (duration: 14m 57s) |
[production] |
| 16:42 |
<akosiaris@cumin1002> |
conftool action : set/pooled=inactive; selector: service=parsoid-php,name=kubernetes.* |
[production] |
| 16:38 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P57743 and previous config saved to /var/cache/conftool/dbconfig/20240222-163806-arnaudb.json |
[production] |
| 16:36 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 16:36 |
<logmsgbot> |
@deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 16:30 |
<fabfur@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp2032.codfw.wmnet,service=(cdn|ats-be) |
[production] |
| 16:30 |
<fabfur@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=cp2031.codfw.wmnet,service=(cdn|ats-be) |
[production] |
| 16:28 |
<fabfur@cumin2002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp[2031-2032].codfw.wmnet |
[production] |
| 16:28 |
<fabfur@cumin2002> |
START - Cookbook sre.hosts.remove-downtime for cp[2031-2032].codfw.wmnet |
[production] |
| 16:28 |
<dancy@deploy2002> |
Started scap: testing T357402 |
[production] |
| 16:26 |
<dancy@deploy2002> |
Installation of scap version "4.66.0" completed for 458 hosts |
[production] |
| 16:25 |
<dancy@deploy2002> |
Installing scap version "4.66.0" for 458 hosts |
[production] |
| 16:23 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P57742 and previous config saved to /var/cache/conftool/dbconfig/20240222-162300-arnaudb.json |
[production] |
| 16:22 |
<volans@cumin1002> |
END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox |
[production] |
| 16:21 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2149 (re)pooling @ 100%: After recloning', diff saved to https://phabricator.wikimedia.org/P57741 and previous config saved to /var/cache/conftool/dbconfig/20240222-162151-root.json |
[production] |
| 16:19 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.reimage for host testvm2002.codfw.wmnet with OS bullseye |
[production] |
| 16:16 |
<volans@cumin1002> |
START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox |
[production] |
| 16:11 |
<mvernon@cumin2002> |
conftool action : set/pooled=true; selector: dnsdisc=swift,name=codfw |
[production] |
| 16:11 |
<Emperor> |
repool codfs-mw T355868 |
[production] |
| 16:10 |
<Emperor> |
repool thanos-fe2002 T355868 |
[production] |
| 16:07 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1214 (T357189)', diff saved to https://phabricator.wikimedia.org/P57740 and previous config saved to /var/cache/conftool/dbconfig/20240222-160753-arnaudb.json |
[production] |
| 16:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2149 (re)pooling @ 75%: After recloning', diff saved to https://phabricator.wikimedia.org/P57739 and previous config saved to /var/cache/conftool/dbconfig/20240222-160646-root.json |
[production] |
| 16:05 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db1214 (T357189)', diff saved to https://phabricator.wikimedia.org/P57738 and previous config saved to /var/cache/conftool/dbconfig/20240222-160534-arnaudb.json |
[production] |
| 16:05 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:05 |
<volans@cumin1002> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts sretest1001.eqiad.wmnet |
[production] |
| 16:05 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:05 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1211 (T357189)', diff saved to https://phabricator.wikimedia.org/P57737 and previous config saved to /var/cache/conftool/dbconfig/20240222-160512-arnaudb.json |
[production] |
| 16:04 |
<volans@cumin1002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts sretest1001.eqiad.wmnet |
[production] |
| 16:00 |
<topranks> |
Commencing network maintenance migrating servers to new switch codfw rack B2 T355868 |
[production] |
| 15:58 |
<cmooney@cumin1002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host testvm2002.codfw.wmnet with OS bullseye |
[production] |
| 15:57 |
<hnowlan> |
depooling mw[1458,1467-1468,1483-1485,1494].eqiad.wmnet in advance of reimaging |
[production] |