|
2023-11-29
ยง
|
| 08:28 |
<marostegui> |
oathauth_users from fishbowl.dblist T348693 |
[production] |
| 08:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 10%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P53934 and previous config saved to /var/cache/conftool/dbconfig/20231129-082748-root.json |
[production] |
| 08:22 |
<marostegui> |
oathauth_users from private.dblist T348693 |
[production] |
| 08:19 |
<marostegui@deploy2002> |
Finished scap: Backport for [[gerrit:978074|Revert "ProductionServices.php: Promote pc1014 to pc2"]] (duration: 08m 01s) |
[production] |
| 08:13 |
<marostegui@deploy2002> |
marostegui: Continuing with sync |
[production] |
| 08:12 |
<marostegui@deploy2002> |
marostegui: Backport for [[gerrit:978074|Revert "ProductionServices.php: Promote pc1014 to pc2"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 5%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P53933 and previous config saved to /var/cache/conftool/dbconfig/20231129-081243-root.json |
[production] |
| 08:11 |
<marostegui@deploy2002> |
Started scap: Backport for [[gerrit:978074|Revert "ProductionServices.php: Promote pc1014 to pc2"]] |
[production] |
| 08:01 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc1012.eqiad.wmnet with OS bookworm |
[production] |
| 07:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'es1027 (re)pooling @ 1%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P53932 and previous config saved to /var/cache/conftool/dbconfig/20231129-075738-root.json |
[production] |
| 07:56 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1027.eqiad.wmnet with OS bookworm |
[production] |
| 07:43 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1012.eqiad.wmnet with reason: host reimage |
[production] |
| 07:40 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1027.eqiad.wmnet with reason: host reimage |
[production] |
| 07:38 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on pc1012.eqiad.wmnet with reason: host reimage |
[production] |
| 07:37 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on es1027.eqiad.wmnet with reason: host reimage |
[production] |
| 07:26 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host pc1012.eqiad.wmnet with OS bookworm |
[production] |
| 07:25 |
<marostegui@deploy2002> |
Finished scap: Backport for [[gerrit:978365|ProductionServices.php: Promote pc1014 to pc2 (T351620)]] (duration: 09m 25s) |
[production] |
| 07:24 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.reimage for host es1027.eqiad.wmnet with OS bookworm |
[production] |
| 07:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool es1027 T351916', diff saved to https://phabricator.wikimedia.org/P53931 and previous config saved to /var/cache/conftool/dbconfig/20231129-072306-root.json |
[production] |
| 07:18 |
<marostegui@deploy2002> |
marostegui: Continuing with sync |
[production] |
| 07:17 |
<marostegui@deploy2002> |
marostegui: Backport for [[gerrit:978365|ProductionServices.php: Promote pc1014 to pc2 (T351620)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 07:15 |
<marostegui@deploy2002> |
Started scap: Backport for [[gerrit:978365|ProductionServices.php: Promote pc1014 to pc2 (T351620)]] |
[production] |
| 07:14 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc2012.codfw.wmnet,pc[1012,1014].eqiad.wmnet with reason: Switch |
[production] |
| 07:13 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on pc2012.codfw.wmnet,pc[1012,1014].eqiad.wmnet with reason: Switch |
[production] |
| 04:33 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1172.eqiad.wmnet with OS bullseye |
[production] |
| 04:33 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1171.eqiad.wmnet with OS bullseye |
[production] |
| 04:33 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1162.eqiad.wmnet with OS bullseye |
[production] |
| 04:32 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1161.eqiad.wmnet with OS bullseye |
[production] |
| 03:13 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-worker1172.eqiad.wmnet with OS bullseye |
[production] |
| 03:13 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-worker1171.eqiad.wmnet with OS bullseye |
[production] |
| 03:12 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-worker1162.eqiad.wmnet with OS bullseye |
[production] |
| 03:12 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.reimage for host an-worker1161.eqiad.wmnet with OS bullseye |
[production] |
| 03:11 |
<jclark@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1164.eqiad.wmnet with OS bullseye |
[production] |
| 03:11 |
<jclark@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" |
[production] |
| 03:10 |
<jclark@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" |
[production] |
| 03:08 |
<jclark@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1168.eqiad.wmnet with OS bullseye |
[production] |
| 03:08 |
<jclark@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" |
[production] |
| 03:06 |
<jclark@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" |
[production] |
| 03:05 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1172.eqiad.wmnet with OS bullseye |
[production] |
| 03:05 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1171.eqiad.wmnet with OS bullseye |
[production] |
| 02:58 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1161.eqiad.wmnet with OS bullseye |
[production] |
| 02:58 |
<jclark@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1162.eqiad.wmnet with OS bullseye |
[production] |
| 02:48 |
<jclark@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1164.eqiad.wmnet with reason: host reimage |
[production] |
| 02:45 |
<jclark@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1168.eqiad.wmnet with reason: host reimage |
[production] |
| 02:43 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1164.eqiad.wmnet with reason: host reimage |
[production] |
| 02:42 |
<jclark@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1168.eqiad.wmnet with reason: host reimage |
[production] |
| 02:33 |
<jclark@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1170.eqiad.wmnet with OS bullseye |
[production] |
| 02:33 |
<jclark@cumin1001> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" |
[production] |
| 02:32 |
<jclark@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1001" |
[production] |
| 02:31 |
<jclark@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1174.eqiad.wmnet with OS bullseye |
[production] |