|
2026-06-16
§
|
| 12:12 |
<cmooney@cumin2002> |
START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2074.codfw.wmnet |
[production] |
| 12:12 |
<cmooney@cumin2002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2051.codfw.wmnet |
[production] |
| 12:07 |
<cmooney@cumin2002> |
START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2051.codfw.wmnet |
[production] |
| 12:06 |
<cmooney@cumin2002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2044.codfw.wmnet |
[production] |
| 12:06 |
<cmooney@cumin2002> |
START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2044.codfw.wmnet |
[production] |
| 12:06 |
<cmooney@cumin2002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet |
[production] |
| 12:05 |
<cmooney@cumin2002> |
START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet |
[production] |
| 12:05 |
<cmooney@cumin2002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2018.codfw.wmnet |
[production] |
| 12:05 |
<cmooney@cumin2002> |
START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2018.codfw.wmnet |
[production] |
| 12:04 |
<cmooney@cumin2002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2017.codfw.wmnet |
[production] |
| 12:04 |
<cmooney@cumin2002> |
START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2017.codfw.wmnet |
[production] |
| 12:04 |
<cmooney@cumin2002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2014.codfw.wmnet |
[production] |
| 12:03 |
<cmooney@cumin2002> |
START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2014.codfw.wmnet |
[production] |
| 12:03 |
<cmooney@cumin2002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2013.codfw.wmnet |
[production] |
| 12:03 |
<cmooney@cumin2002> |
START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2013.codfw.wmnet |
[production] |
| 12:02 |
<cmooney@cumin2002> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2012.codfw.wmnet |
[production] |
| 12:02 |
<cmooney@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2001.codfw.wmnet |
[production] |
| 12:01 |
<cmooney@cumin2002> |
START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2012.codfw.wmnet |
[production] |
| 11:57 |
<cmooney@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2001.codfw.wmnet |
[production] |
| 11:49 |
<cmooney@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2176: codfw rack a5 depool for switch maintenance T428020 |
[production] |
| 11:49 |
<cmooney@cumin1003> |
START - Cookbook sre.mysql.depool depool db2176: codfw rack a5 depool for switch maintenance T428020 |
[production] |
| 11:49 |
<cmooney@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2175: codfw rack a5 depool for switch maintenance T428020 |
[production] |
| 11:48 |
<cmooney@cumin1003> |
START - Cookbook sre.mysql.depool depool db2175: codfw rack a5 depool for switch maintenance T428020 |
[production] |
| 11:48 |
<cmooney@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2157: codfw rack a5 depool for switch maintenance T428020 |
[production] |
| 11:48 |
<cmooney@cumin1003> |
START - Cookbook sre.mysql.depool depool db2157: codfw rack a5 depool for switch maintenance T428020 |
[production] |
| 11:48 |
<cmooney@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2154: codfw rack a5 depool for switch maintenance T428020 |
[production] |
| 11:47 |
<cmooney@cumin1003> |
START - Cookbook sre.mysql.depool depool db2154: codfw rack a5 depool for switch maintenance T428020 |
[production] |
| 11:46 |
<cmooney@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2153: codfw rack a5 depool for switch maintenance T428020 |
[production] |
| 11:46 |
<cmooney@cumin1003> |
START - Cookbook sre.mysql.depool depool db2153: codfw rack a5 depool for switch maintenance T428020 |
[production] |
| 10:49 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool es1037 T429118', diff saved to https://phabricator.wikimedia.org/P94172 and previous config saved to /var/cache/conftool/dbconfig/20260616-104931-marostegui.json |
[production] |
| 09:34 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2035: Upgrading es2035.codfw.wmnet |
[production] |
| 09:34 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.depool depool es2035: Upgrading es2035.codfw.wmnet |
[production] |
| 09:32 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool es2035 T429303', diff saved to https://phabricator.wikimedia.org/P94164 and previous config saved to /var/cache/conftool/dbconfig/20260616-093247-marostegui.json |
[production] |
| 08:47 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1036: Upgrading es1036.eqiad.wmnet |
[production] |
| 08:45 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.depool depool es1036: Upgrading es1036.eqiad.wmnet |
[production] |
| 06:24 |
<marostegui@cumin1003> |
END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es1047: Upgrading es1047.eqiad.wmnet |
[production] |
| 05:59 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.depool depool es1047: Upgrading es1047.eqiad.wmnet |
[production] |
|
2026-06-15
§
|
| 15:02 |
<topranks> |
depool esams due to cr2-esams rpd crash |
[production] |
| 15:02 |
<cmooney@cumin1003> |
END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: no reason specified, no task ID specified] |
[production] |
| 15:01 |
<cmooney@cumin1003> |
START - Cookbook sre.dns.admin DNS admin: depool esams [reason: no reason specified, no task ID specified] |
[production] |
| 14:01 |
<cwilliams@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1196: Upgrading db1196.eqiad.wmnet |
[production] |
| 14:00 |
<cwilliams@cumin1003> |
START - Cookbook sre.mysql.depool depool db1196: Upgrading db1196.eqiad.wmnet |
[production] |
| 11:50 |
<cwilliams@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2216: Upgrading db2216.codfw.wmnet |
[production] |
| 11:49 |
<cwilliams@cumin1003> |
START - Cookbook sre.mysql.depool depool db2216: Upgrading db2216.codfw.wmnet |
[production] |
| 11:46 |
<cwilliams@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1251: Upgrading db1251.eqiad.wmnet |
[production] |
| 11:45 |
<cwilliams@cumin1003> |
START - Cookbook sre.mysql.depool depool db1251: Upgrading db1251.eqiad.wmnet |
[production] |
| 10:59 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2036: Upgrading es2036.codfw.wmnet |
[production] |
| 10:58 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.depool depool es2036: Upgrading es2036.codfw.wmnet |
[production] |
| 08:53 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2037: Upgrading es2037.codfw.wmnet |
[production] |
| 08:53 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.depool depool es2037: Upgrading es2037.codfw.wmnet |
[production] |