5901-5950 of 10000 results (9ms)
2026-06-10 §
13:33 <cwilliams@cumin1003> START - Cookbook sre.mysql.depool depool db1195: Upgrading db1195.eqiad.wmnet [production]
11:49 <cwilliams@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2170: Upgrading db2170.codfw.wmnet [production]
11:49 <cwilliams@cumin1003> START - Cookbook sre.mysql.depool depool db2170: Upgrading db2170.codfw.wmnet [production]
11:46 <cwilliams@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1186: Upgrading db1186.eqiad.wmnet [production]
11:45 <cwilliams@cumin1003> START - Cookbook sre.mysql.depool depool db1186: Upgrading db1186.eqiad.wmnet [production]
11:09 <root@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1151: Security updates [production]
11:09 <root@cumin1003> START - Cookbook sre.mysql.depool depool db1151: Security updates [production]
10:27 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet [production]
10:14 <root@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1152: Security updates [production]
10:14 <root@cumin1003> START - Cookbook sre.mysql.depool depool db1152: Security updates [production]
09:22 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet [production]
06:45 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet [production]
06:45 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet [production]
05:43 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet [production]
05:43 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet [production]
05:42 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet [production]
05:41 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet [production]
2026-06-09 §
14:06 <ayounsi@cumin1003> END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: rack depool [production]
13:20 <ayounsi@cumin1003> START - Cookbook sre.mysql.pool pool db2241: rack depool [production]
13:17 <ayounsi@cumin1003> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2006.codfw.wmnet [production]
13:17 <ayounsi@cumin1003> START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2006.codfw.wmnet [production]
13:16 <ayounsi@cumin1003> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2251-2253].codfw.wmnet [production]
13:16 <ayounsi@cumin1003> START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2251-2253].codfw.wmnet [production]
13:16 <ayounsi@cumin1003> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2005.codfw.wmnet [production]
13:16 <ayounsi@cumin1003> START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2005.codfw.wmnet [production]
13:11 <ayounsi@cumin1003> END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 [production]
12:57 <ayounsi@cumin1003> START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 [production]
12:46 <ayounsi@cumin1003> END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 [production]
12:45 <ayounsi@cumin1003> START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 [production]
12:34 <cwilliams@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2153: Upgrading db2153.codfw.wmnet [production]
12:34 <cwilliams@cumin1003> START - Cookbook sre.mysql.depool depool db2153: Upgrading db2153.codfw.wmnet [production]
12:33 <cwilliams@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1184: Upgrading db1184.eqiad.wmnet [production]
12:33 <cwilliams@cumin1003> START - Cookbook sre.mysql.depool depool db1184: Upgrading db1184.eqiad.wmnet [production]
12:27 <ayounsi@cumin1003> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2005.codfw.wmnet [production]
12:24 <ayounsi@cumin1003> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2006.codfw.wmnet [production]
12:23 <ayounsi@cumin1003> START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2006.codfw.wmnet [production]
12:22 <ayounsi@cumin1003> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2251-2253].codfw.wmnet [production]
12:22 <ayounsi@cumin1003> START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2005.codfw.wmnet [production]
12:20 <ayounsi@cumin1003> START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2251-2253].codfw.wmnet [production]
12:20 <ayounsi@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: rack depool [production]
12:20 <ayounsi@cumin1003> START - Cookbook sre.mysql.depool depool db2241: rack depool [production]
12:17 <ayounsi@cumin1003> END (FAIL) - Cookbook sre.network.depool-rack (exit_code=99) with action 'depool' for codfw rack A4 [production]
12:16 <ayounsi@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 24 hosts with reason: Rack A4 depool [production]
12:14 <ayounsi@cumin1003> START - Cookbook sre.network.depool-rack with action 'depool' for codfw rack A4 [production]
10:53 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2046: Upgrading es2046.codfw.wmnet [production]
10:53 <marostegui@cumin1003> START - Cookbook sre.mysql.depool depool es2046: Upgrading es2046.codfw.wmnet [production]
09:48 <marostegui@cumin1003> END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) depool es1043: Upgrading es1043.eqiad.wmnet [production]
09:14 <marostegui@cumin1003> START - Cookbook sre.mysql.depool depool es1043: Upgrading es1043.eqiad.wmnet [production]
06:24 <fceratto@cumin1003> dbctl commit (dc=all): 'Depool db1160 T426086', diff saved to https://phabricator.wikimedia.org/P93940 and previous config saved to /var/cache/conftool/dbconfig/20260609-062412-fceratto.json [production]
05:27 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1237: Upgrading db1237.eqiad.wmnet [production]