|
2023-01-09
ยง
|
| 22:32 |
<bking@cumin1001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.UPGRADE (3 nodes at a time) for ElasticSearch cluster search_codfw: plugin upgrade - bking@cumin1001 - T324247 |
[production] |
| 22:28 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host mc2043.codfw.wmnet |
[production] |
| 22:25 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc2030.codfw.wmnet |
[production] |
| 22:25 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:25 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc2030.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1001" |
[production] |
| 22:15 |
<jiji@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc2030.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1001" |
[production] |
| 22:11 |
<jiji@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:05 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mc2030.codfw.wmnet |
[production] |
| 22:03 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc2029.codfw.wmnet |
[production] |
| 22:03 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:03 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc2029.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1001" |
[production] |
| 22:00 |
<jiji@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc2029.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1001" |
[production] |
| 21:54 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2042.codfw.wmnet |
[production] |
| 21:52 |
<kindrobot> |
close UTC late backport window |
[production] |
| 21:50 |
<jiji@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 21:47 |
<kindrobot@deploy1002> |
Sync cancelled. |
[production] |
| 21:47 |
<kindrobot@deploy1002> |
kindrobot and trainbranchbot: Backport for [[gerrit:877260|Revert "[config]: Deploy GDI Safety Survey Wave 4"]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 21:47 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host mc2042.codfw.wmnet |
[production] |
| 21:45 |
<bking@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (3 nodes at a time) for ElasticSearch cluster search_codfw: plugin upgrade - bking@cumin1001 - T324247 |
[production] |
| 21:45 |
<kindrobot@deploy1002> |
Started scap: Backport for [[gerrit:877260|Revert "[config]: Deploy GDI Safety Survey Wave 4"]] |
[production] |
| 21:39 |
<kindrobot@deploy1002> |
Sync cancelled. |
[production] |
| 21:38 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mc2029.codfw.wmnet |
[production] |
| 21:37 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc2027.codfw.wmnet |
[production] |
| 21:37 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 21:37 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc2027.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1001" |
[production] |
| 21:34 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic plugin upgrade - bking@cumin1001 - T324247 |
[production] |
| 21:29 |
<jiji@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc2027.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1001" |
[production] |
| 21:27 |
<jiji@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 21:26 |
<kindrobot@deploy1002> |
kindrobot and essexigyan: Backport for [[gerrit:877197|[config]: Deploy GDI Safety Survey Wave 4 (T325136)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet |
[production] |
| 21:24 |
<kindrobot@deploy1002> |
Started scap: Backport for [[gerrit:877197|[config]: Deploy GDI Safety Survey Wave 4 (T325136)]] |
[production] |
| 21:21 |
<kindrobot> |
starting UTC late backport window |
[production] |
| 21:21 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mc2027.codfw.wmnet |
[production] |
| 21:18 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc2026.codfw.wmnet |
[production] |
| 21:18 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 21:18 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc2026.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1001" |
[production] |
| 21:09 |
<jiji@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc2026.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1001" |
[production] |
| 21:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1143', diff saved to https://phabricator.wikimedia.org/P42936 and previous config saved to /var/cache/conftool/dbconfig/20230109-210940-marostegui.json |
[production] |
| 21:04 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc2041.codfw.wmnet |
[production] |
| 21:03 |
<jiji@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 20:57 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host mc2041.codfw.wmnet |
[production] |
| 20:57 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts mc2026.codfw.wmnet |
[production] |
| 20:52 |
<bking@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic plugin upgrade - bking@cumin1001 - T324247 |
[production] |
| 20:52 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge plugin upgrade - bking@cumin1001 - T324247 |
[production] |
| 20:44 |
<bking@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge plugin upgrade - bking@cumin1001 - T324247 |
[production] |
| 20:44 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge plugin upgrade - bking@cumin1001 - T324247 |
[production] |
| 20:44 |
<bking@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge plugin upgrade - bking@cumin1001 - T324247 |
[production] |
| 20:36 |
<Amir1> |
deleting global usage coming from commons in commons (T322588) |
[production] |
| 20:36 |
<bking@cumin1001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge plugin upgrade - bking@cumin1001 - T324247 |
[production] |
| 20:35 |
<bking@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.UPGRADE (1 nodes at a time) for ElasticSearch cluster relforge: relforge plugin upgrade - bking@cumin1001 - T324247 |
[production] |
| 20:34 |
<bd808@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply |
[production] |