|
2025-06-19
§
|
| 14:38 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2021.codfw.wmnet |
[production] |
| 12:01 |
<jayme@cumin1002> |
END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in codfw/codfw: maintenance |
[production] |
| 11:46 |
<jayme@cumin1002> |
START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/codfw: maintenance |
[production] |
| 10:46 |
<jayme@cumin1002> |
END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in codfw/codfw: maintenance |
[production] |
| 10:31 |
<jayme@cumin1002> |
START - Cookbook sre.k8s.pool-depool-cluster depool all services in codfw/codfw: maintenance |
[production] |
| 10:19 |
<Emperor> |
depool / restart / repool ms-fe1009 [some idle timeouts] |
[production] |
| 09:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2196', diff saved to https://phabricator.wikimedia.org/P78422 and previous config saved to /var/cache/conftool/dbconfig/20250619-091532-root.json |
[production] |
| 08:56 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2021.codfw.wmnet |
[production] |
| 08:55 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2021.codfw.wmnet |
[production] |
| 08:50 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2021.codfw.wmnet |
[production] |
| 07:55 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1179', diff saved to https://phabricator.wikimedia.org/P78411 and previous config saved to /var/cache/conftool/dbconfig/20250619-075548-root.json |
[production] |
| 05:38 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2186', diff saved to https://phabricator.wikimedia.org/P78394 and previous config saved to /var/cache/conftool/dbconfig/20250619-053826-root.json |
[production] |
| 05:07 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool pc2 T378715', diff saved to https://phabricator.wikimedia.org/P78392 and previous config saved to /var/cache/conftool/dbconfig/20250619-050725-root.json |
[production] |
|
2025-06-18
§
|
| 17:59 |
<jmm@cumin1003> |
END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet |
[production] |
| 17:52 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depool db2155 for queries (T385167)', diff saved to https://phabricator.wikimedia.org/P78379 and previous config saved to /var/cache/conftool/dbconfig/20250618-175206-ladsgroup.json |
[production] |
| 14:48 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet |
[production] |
| 14:47 |
<jmm@cumin1003> |
END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet |
[production] |
| 12:45 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet |
[production] |
| 12:43 |
<jmm@cumin1003> |
END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet |
[production] |
| 11:13 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet |
[production] |
| 10:58 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2030.codfw.wmnet |
[production] |
| 10:47 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2030.codfw.wmnet |
[production] |
| 10:40 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2191', diff saved to https://phabricator.wikimedia.org/P78338 and previous config saved to /var/cache/conftool/dbconfig/20250618-104033-root.json |
[production] |
| 10:10 |
<jayme@cumin1002> |
END (FAIL) - Cookbook sre.k8s.pool-depool-cluster (exit_code=99) depool 44 services in codfw/codfw: pre-upgrade-test |
[production] |
| 10:10 |
<jayme@cumin1002> |
START - Cookbook sre.k8s.pool-depool-cluster depool 44 services in codfw/codfw: pre-upgrade-test |
[production] |
| 07:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2231', diff saved to https://phabricator.wikimedia.org/P78304 and previous config saved to /var/cache/conftool/dbconfig/20250618-072404-root.json |
[production] |
| 07:12 |
<jayme@cumin1002> |
END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) check 44 services in codfw: maintenance |
[production] |
| 07:12 |
<jayme@cumin1002> |
START - Cookbook sre.k8s.pool-depool-cluster check 44 services in codfw: maintenance |
[production] |
| 07:02 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1156', diff saved to https://phabricator.wikimedia.org/P78301 and previous config saved to /var/cache/conftool/dbconfig/20250618-070239-root.json |
[production] |
| 05:32 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1188', diff saved to https://phabricator.wikimedia.org/P78286 and previous config saved to /var/cache/conftool/dbconfig/20250618-053253-root.json |
[production] |
| 05:19 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1173 T397198', diff saved to https://phabricator.wikimedia.org/P78283 and previous config saved to /var/cache/conftool/dbconfig/20250618-051935-root.json |
[production] |
|
2025-06-17
§
|
| 16:47 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2024.codfw.wmnet |
[production] |
| 13:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1165 T395989', diff saved to https://phabricator.wikimedia.org/P78191 and previous config saved to /var/cache/conftool/dbconfig/20250617-135706-marostegui.json |
[production] |
| 12:28 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2024.codfw.wmnet |
[production] |
| 12:25 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2024.codfw.wmnet |
[production] |
| 12:21 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2024.codfw.wmnet |
[production] |
| 12:06 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2023.codfw.wmnet |
[production] |
| 07:11 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2023.codfw.wmnet |
[production] |
| 07:11 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2023.codfw.wmnet |
[production] |
| 07:10 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2023.codfw.wmnet |
[production] |
| 06:34 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2023.codfw.wmnet |
[production] |
| 06:30 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2023.codfw.wmnet |
[production] |
| 06:14 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool es1030', diff saved to https://phabricator.wikimedia.org/P78118 and previous config saved to /var/cache/conftool/dbconfig/20250617-061441-marostegui.json |
[production] |
| 06:06 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool es1029', diff saved to https://phabricator.wikimedia.org/P78114 and previous config saved to /var/cache/conftool/dbconfig/20250617-060640-marostegui.json |
[production] |
| 06:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool es1028', diff saved to https://phabricator.wikimedia.org/P78112 and previous config saved to /var/cache/conftool/dbconfig/20250617-060347-marostegui.json |
[production] |