8851-8900 of 10000 results (10ms)
2025-06-23 §
09:06 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1220', diff saved to https://phabricator.wikimedia.org/P78599 and previous config saved to /var/cache/conftool/dbconfig/20250623-090619-root.json [production]
08:39 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1224', diff saved to https://phabricator.wikimedia.org/P78594 and previous config saved to /var/cache/conftool/dbconfig/20250623-083954-root.json [production]
06:30 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es2039 T397599', diff saved to https://phabricator.wikimedia.org/P78568 and previous config saved to /var/cache/conftool/dbconfig/20250623-063050-marostegui.json [production]
06:15 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es2035 T397597', diff saved to https://phabricator.wikimedia.org/P78562 and previous config saved to /var/cache/conftool/dbconfig/20250623-061511-marostegui.json [production]
05:47 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2215 T397419', diff saved to https://phabricator.wikimedia.org/P78556 and previous config saved to /var/cache/conftool/dbconfig/20250623-054725-marostegui.json [production]
2025-06-19 §
14:38 <jmm@cumin1003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2021.codfw.wmnet [production]
12:01 <jayme@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) pool all services in codfw/codfw: maintenance [production]
11:46 <jayme@cumin1002> START - Cookbook sre.k8s.pool-depool-cluster pool all services in codfw/codfw: maintenance [production]
10:46 <jayme@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) depool all services in codfw/codfw: maintenance [production]
10:31 <jayme@cumin1002> START - Cookbook sre.k8s.pool-depool-cluster depool all services in codfw/codfw: maintenance [production]
10:19 <Emperor> depool / restart / repool ms-fe1009 [some idle timeouts] [production]
09:15 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2196', diff saved to https://phabricator.wikimedia.org/P78422 and previous config saved to /var/cache/conftool/dbconfig/20250619-091532-root.json [production]
08:56 <jmm@cumin1003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2021.codfw.wmnet [production]
08:55 <jmm@cumin1003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2021.codfw.wmnet [production]
08:50 <jmm@cumin1003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2021.codfw.wmnet [production]
07:55 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1179', diff saved to https://phabricator.wikimedia.org/P78411 and previous config saved to /var/cache/conftool/dbconfig/20250619-075548-root.json [production]
05:38 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2186', diff saved to https://phabricator.wikimedia.org/P78394 and previous config saved to /var/cache/conftool/dbconfig/20250619-053826-root.json [production]
05:07 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool pc2 T378715', diff saved to https://phabricator.wikimedia.org/P78392 and previous config saved to /var/cache/conftool/dbconfig/20250619-050725-root.json [production]
2025-06-18 §
17:59 <jmm@cumin1003> END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet [production]
17:52 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depool db2155 for queries (T385167)', diff saved to https://phabricator.wikimedia.org/P78379 and previous config saved to /var/cache/conftool/dbconfig/20250618-175206-ladsgroup.json [production]
14:48 <jmm@cumin1003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet [production]
14:47 <jmm@cumin1003> END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet [production]
12:45 <jmm@cumin1003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet [production]
12:43 <jmm@cumin1003> END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti1048.eqiad.wmnet [production]
11:13 <jmm@cumin1003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1048.eqiad.wmnet [production]
10:58 <jmm@cumin1003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2030.codfw.wmnet [production]
10:47 <jmm@cumin1003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2030.codfw.wmnet [production]
10:40 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2191', diff saved to https://phabricator.wikimedia.org/P78338 and previous config saved to /var/cache/conftool/dbconfig/20250618-104033-root.json [production]
10:10 <jayme@cumin1002> END (FAIL) - Cookbook sre.k8s.pool-depool-cluster (exit_code=99) depool 44 services in codfw/codfw: pre-upgrade-test [production]
10:10 <jayme@cumin1002> START - Cookbook sre.k8s.pool-depool-cluster depool 44 services in codfw/codfw: pre-upgrade-test [production]
07:24 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2231', diff saved to https://phabricator.wikimedia.org/P78304 and previous config saved to /var/cache/conftool/dbconfig/20250618-072404-root.json [production]
07:12 <jayme@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-cluster (exit_code=0) check 44 services in codfw: maintenance [production]
07:12 <jayme@cumin1002> START - Cookbook sre.k8s.pool-depool-cluster check 44 services in codfw: maintenance [production]
07:02 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1156', diff saved to https://phabricator.wikimedia.org/P78301 and previous config saved to /var/cache/conftool/dbconfig/20250618-070239-root.json [production]
05:32 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1188', diff saved to https://phabricator.wikimedia.org/P78286 and previous config saved to /var/cache/conftool/dbconfig/20250618-053253-root.json [production]
05:19 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1173 T397198', diff saved to https://phabricator.wikimedia.org/P78283 and previous config saved to /var/cache/conftool/dbconfig/20250618-051935-root.json [production]
2025-06-17 §
16:47 <jmm@cumin1003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2024.codfw.wmnet [production]
13:57 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1165 T395989', diff saved to https://phabricator.wikimedia.org/P78191 and previous config saved to /var/cache/conftool/dbconfig/20250617-135706-marostegui.json [production]
12:28 <jmm@cumin1003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2024.codfw.wmnet [production]
12:25 <jmm@cumin1003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2024.codfw.wmnet [production]
12:21 <jmm@cumin1003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2024.codfw.wmnet [production]
12:06 <jmm@cumin1003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2023.codfw.wmnet [production]
07:11 <jmm@cumin1003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2023.codfw.wmnet [production]
07:11 <jmm@cumin1003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2023.codfw.wmnet [production]
07:10 <jmm@cumin1003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2023.codfw.wmnet [production]
06:34 <jmm@cumin1003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2023.codfw.wmnet [production]
06:30 <jmm@cumin1003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2023.codfw.wmnet [production]
06:14 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es1030', diff saved to https://phabricator.wikimedia.org/P78118 and previous config saved to /var/cache/conftool/dbconfig/20250617-061441-marostegui.json [production]
06:06 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es1029', diff saved to https://phabricator.wikimedia.org/P78114 and previous config saved to /var/cache/conftool/dbconfig/20250617-060640-marostegui.json [production]
06:03 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es1028', diff saved to https://phabricator.wikimedia.org/P78112 and previous config saved to /var/cache/conftool/dbconfig/20250617-060347-marostegui.json [production]