8551-8600 of 10000 results (9ms)
2025-09-23 §
16:03 <jasmine@cumin1003> START - Cookbook sre.discovery.datacenter depool all services in eqiad: Moving services to codfw, Southward DC Switchover Day 1 - T399891 [production]
16:03 <jasmine@cumin1003> END (ERROR) - Cookbook sre.discovery.datacenter (exit_code=93) depool all services in eqiad: Moving services to codfw, Southward DC Switchover Day 1 - T399891 [production]
15:52 <jasmine@cumin1003> START - Cookbook sre.discovery.datacenter depool all services in eqiad: Moving services to codfw, Southward DC Switchover Day 1 - T399891 [production]
15:30 <jasmine@cumin1003> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site eqiad [reason: Moving traffic to codfw, Southward DC Switchover Day 1, T399891] [production]
15:30 <jasmine@cumin1003> START - Cookbook sre.dns.admin DNS admin: depool site eqiad [reason: Moving traffic to codfw, Southward DC Switchover Day 1, T399891] [production]
14:06 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7004.magru.wmnet [production]
13:52 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7004.magru.wmnet [production]
13:45 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7003.magru.wmnet [production]
13:34 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7003.magru.wmnet [production]
13:31 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7001.magru.wmnet [production]
13:19 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7001.magru.wmnet [production]
12:57 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2034.codfw.wmnet [production]
12:30 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet [production]
12:03 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2033.codfw.wmnet [production]
11:54 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2033.codfw.wmnet [production]
11:49 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti-test2001.codfw.wmnet [production]
11:39 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti-test2001.codfw.wmnet [production]
2025-09-22 §
09:27 <klausman@cumin2002> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool inference in eqiad: maintenance [production]
09:22 <klausman@cumin2002> START - Cookbook sre.discovery.service-route depool inference in eqiad: maintenance [production]
2025-09-19 §
14:05 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1043.eqiad.wmnet [production]
10:12 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2033.codfw.wmnet [production]
10:09 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2033.codfw.wmnet [production]
09:44 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1043.eqiad.wmnet [production]
09:43 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1043.eqiad.wmnet [production]
09:41 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1043.eqiad.wmnet [production]
2025-09-18 §
10:59 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depool db1203', diff saved to https://phabricator.wikimedia.org/P83426 and previous config saved to /var/cache/conftool/dbconfig/20250918-105922-ladsgroup.json [production]
10:59 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depool db1192', diff saved to https://phabricator.wikimedia.org/P83425 and previous config saved to /var/cache/conftool/dbconfig/20250918-105905-ladsgroup.json [production]
06:34 <jynus@cumin1003> dbctl commit (dc=all): 'Depool es2027 T404940', diff saved to https://phabricator.wikimedia.org/P83420 and previous config saved to /var/cache/conftool/dbconfig/20250918-063436-jynus.json [production]
2025-09-17 §
13:44 <fceratto@cumin1002> END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) es2027 - Depool es2027.codfw.wmnet to then clone it to es2050.codfw.wmnet - fceratto@cumin1002 [production]
13:35 <fceratto@cumin1002> START - Cookbook sre.mysql.depool es2027 - Depool es2027.codfw.wmnet to then clone it to es2050.codfw.wmnet - fceratto@cumin1002 [production]
13:34 <fceratto@cumin1002> dbctl commit (dc=all): 'Depool es2027 T402859', diff saved to https://phabricator.wikimedia.org/P83408 and previous config saved to /var/cache/conftool/dbconfig/20250917-133454-fceratto.json [production]
13:29 <fceratto@cumin1002> END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) es2027 - Depool for cloning [production]
13:29 <fceratto@cumin1002> START - Cookbook sre.mysql.depool es2027 - Depool for cloning [production]
13:03 <fceratto@cumin1002> END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) es2027 - Depool es2027.codfw.wmnet to then clone it to es2050.codfw.wmnet - fceratto@cumin1002 [production]
13:03 <fceratto@cumin1002> START - Cookbook sre.mysql.depool es2027 - Depool es2027.codfw.wmnet to then clone it to es2050.codfw.wmnet - fceratto@cumin1002 [production]
13:03 <fceratto@cumin1002> END (FAIL) - Cookbook sre.mysql.depool (exit_code=99) es2027 - Depool es2027.codfw.wmnet to then clone it to es2050.codfw.wmnet - fceratto@cumin1002 [production]
13:01 <fceratto@cumin1002> START - Cookbook sre.mysql.depool es2027 - Depool es2027.codfw.wmnet to then clone it to es2050.codfw.wmnet - fceratto@cumin1002 [production]
2025-09-15 §
14:24 <fceratto@cumin1002> dbctl commit (dc=all): 'Depool db2220 T404595', diff saved to https://phabricator.wikimedia.org/P83333 and previous config saved to /var/cache/conftool/dbconfig/20250915-142436-fceratto.json [production]
13:33 <fceratto@cumin1002> dbctl commit (dc=all): 'Depool db2229 T404586', diff saved to https://phabricator.wikimedia.org/P83322 and previous config saved to /var/cache/conftool/dbconfig/20250915-133322-fceratto.json [production]
2025-09-11 §
13:32 <vgutierrez> depool cp7001 [production]
11:15 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depool db1184 T404326', diff saved to https://phabricator.wikimedia.org/P83239 and previous config saved to /var/cache/conftool/dbconfig/20250911-111545-ladsgroup.json [production]
10:22 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depool db1160 T404274', diff saved to https://phabricator.wikimedia.org/P83233 and previous config saved to /var/cache/conftool/dbconfig/20250911-102232-ladsgroup.json [production]
09:16 <fceratto@cumin1002> dbctl commit (dc=all): 'Depool db2179 T404299', diff saved to https://phabricator.wikimedia.org/P83226 and previous config saved to /var/cache/conftool/dbconfig/20250911-091626-fceratto.json [production]
2025-09-10 §
23:44 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depool db1209 T404277', diff saved to https://phabricator.wikimedia.org/P83213 and previous config saved to /var/cache/conftool/dbconfig/20250910-234456-ladsgroup.json [production]
10:56 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depool db1236 T404180', diff saved to https://phabricator.wikimedia.org/P83107 and previous config saved to /var/cache/conftool/dbconfig/20250910-105650-ladsgroup.json [production]
08:19 <elukey@cumin1003> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1011.eqiad.wmnet [production]
08:19 <elukey@cumin1003> START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1011.eqiad.wmnet [production]
08:12 <elukey@cumin1003> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1011.eqiad.wmnet [production]
08:06 <elukey@cumin1003> START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1011.eqiad.wmnet [production]
08:06 <elukey@cumin1003> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1010.eqiad.wmnet [production]