|
2026-06-10
§
|
| 13:33 |
<cwilliams@cumin1003> |
START - Cookbook sre.mysql.depool depool db1195: Upgrading db1195.eqiad.wmnet |
[production] |
| 11:49 |
<cwilliams@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2170: Upgrading db2170.codfw.wmnet |
[production] |
| 11:49 |
<cwilliams@cumin1003> |
START - Cookbook sre.mysql.depool depool db2170: Upgrading db2170.codfw.wmnet |
[production] |
| 11:46 |
<cwilliams@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1186: Upgrading db1186.eqiad.wmnet |
[production] |
| 11:45 |
<cwilliams@cumin1003> |
START - Cookbook sre.mysql.depool depool db1186: Upgrading db1186.eqiad.wmnet |
[production] |
| 11:09 |
<root@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1151: Security updates |
[production] |
| 11:09 |
<root@cumin1003> |
START - Cookbook sre.mysql.depool depool db1151: Security updates |
[production] |
| 10:27 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet |
[production] |
| 10:14 |
<root@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1152: Security updates |
[production] |
| 10:14 |
<root@cumin1003> |
START - Cookbook sre.mysql.depool depool db1152: Security updates |
[production] |
| 09:22 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet |
[production] |
| 06:45 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet |
[production] |
| 06:45 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet |
[production] |
| 05:43 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet |
[production] |
| 05:43 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet |
[production] |
| 05:42 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet |
[production] |
| 05:41 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet |
[production] |
|
2026-06-09
§
|
| 14:06 |
<ayounsi@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: rack depool |
[production] |
| 13:20 |
<ayounsi@cumin1003> |
START - Cookbook sre.mysql.pool pool db2241: rack depool |
[production] |
| 13:17 |
<ayounsi@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2006.codfw.wmnet |
[production] |
| 13:17 |
<ayounsi@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2006.codfw.wmnet |
[production] |
| 13:16 |
<ayounsi@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2251-2253].codfw.wmnet |
[production] |
| 13:16 |
<ayounsi@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2251-2253].codfw.wmnet |
[production] |
| 13:16 |
<ayounsi@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2005.codfw.wmnet |
[production] |
| 13:16 |
<ayounsi@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2005.codfw.wmnet |
[production] |
| 13:11 |
<ayounsi@cumin1003> |
END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 |
[production] |
| 12:57 |
<ayounsi@cumin1003> |
START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 |
[production] |
| 12:46 |
<ayounsi@cumin1003> |
END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 |
[production] |
| 12:45 |
<ayounsi@cumin1003> |
START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 |
[production] |
| 12:34 |
<cwilliams@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2153: Upgrading db2153.codfw.wmnet |
[production] |
| 12:34 |
<cwilliams@cumin1003> |
START - Cookbook sre.mysql.depool depool db2153: Upgrading db2153.codfw.wmnet |
[production] |
| 12:33 |
<cwilliams@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1184: Upgrading db1184.eqiad.wmnet |
[production] |
| 12:33 |
<cwilliams@cumin1003> |
START - Cookbook sre.mysql.depool depool db1184: Upgrading db1184.eqiad.wmnet |
[production] |
| 12:27 |
<ayounsi@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2005.codfw.wmnet |
[production] |
| 12:24 |
<ayounsi@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2006.codfw.wmnet |
[production] |
| 12:23 |
<ayounsi@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2006.codfw.wmnet |
[production] |
| 12:22 |
<ayounsi@cumin1003> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2251-2253].codfw.wmnet |
[production] |
| 12:22 |
<ayounsi@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2005.codfw.wmnet |
[production] |
| 12:20 |
<ayounsi@cumin1003> |
START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2251-2253].codfw.wmnet |
[production] |
| 12:20 |
<ayounsi@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: rack depool |
[production] |
| 12:20 |
<ayounsi@cumin1003> |
START - Cookbook sre.mysql.depool depool db2241: rack depool |
[production] |
| 12:17 |
<ayounsi@cumin1003> |
END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 |
[production] |
| 12:16 |
<ayounsi@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 24 hosts with reason: Rack A4 depool |
[production] |
| 12:14 |
<ayounsi@cumin1003> |
START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 |
[production] |
| 10:53 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2046: Upgrading es2046.codfw.wmnet |
[production] |
| 10:53 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.depool depool es2046: Upgrading es2046.codfw.wmnet |
[production] |
| 09:48 |
<marostegui@cumin1003> |
END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es1043: Upgrading es1043.eqiad.wmnet |
[production] |
| 09:14 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.depool depool es1043: Upgrading es1043.eqiad.wmnet |
[production] |
| 06:24 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Depool db1160 T426086', diff saved to https://phabricator.wikimedia.org/P93940 and previous config saved to /var/cache/conftool/dbconfig/20260609-062412-fceratto.json |
[production] |
| 05:27 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1237: Upgrading db1237.eqiad.wmnet |
[production] |