501-550 of 10000 results (29ms)
2026-03-04 §
02:03 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
02:03 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
02:00 <mwpresync@deploy2002> Started scap build-images: Publishing wmf/next image [production]
01:57 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
01:57 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
01:57 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
01:56 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1263 (T418465)', diff saved to https://phabricator.wikimedia.org/P89793 and previous config saved to /var/cache/conftool/dbconfig/20260304-015657-marostegui.json [production]
01:53 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
01:52 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
01:50 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) [toolsbeta]
01:49 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster [toolsbeta]
01:41 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P89792 and previous config saved to /var/cache/conftool/dbconfig/20260304-014150-marostegui.json [production]
01:26 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P89791 and previous config saved to /var/cache/conftool/dbconfig/20260304-012642-marostegui.json [production]
01:23 <zabe@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply [production]
01:22 <zabe@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-experimental: apply [production]
01:19 <wm-bot2> Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22650516003 (https://github.com/cluebotng/component-configs/commits/e7a1e2e06f2ccf038c06cb203369f336c298cf6c) [tools.cluebotng-review]
01:19 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) [toolsbeta]
01:18 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster [toolsbeta]
01:13 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) [toolsbeta]
01:12 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster [toolsbeta]
01:11 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1263 (T418465)', diff saved to https://phabricator.wikimedia.org/P89790 and previous config saved to /var/cache/conftool/dbconfig/20260304-011134-marostegui.json [production]
00:53 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) [toolsbeta]
00:52 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster [toolsbeta]
00:46 <marostegui@cumin1003> dbctl commit (dc=all): 'Depooling db1263 (T418465)', diff saved to https://phabricator.wikimedia.org/P89789 and previous config saved to /var/cache/conftool/dbconfig/20260304-004638-marostegui.json [production]
00:46 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) [toolsbeta]
00:46 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1263.eqiad.wmnet with reason: Maintenance [production]
00:46 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1262 (T418465)', diff saved to https://phabricator.wikimedia.org/P89788 and previous config saved to /var/cache/conftool/dbconfig/20260304-004615-marostegui.json [production]
00:45 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster [toolsbeta]
00:45 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) [toolsbeta]
00:44 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster [toolsbeta]
00:40 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) [toolsbeta]
00:39 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster [toolsbeta]
00:38 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) [toolsbeta]
00:37 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster [toolsbeta]
00:37 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) [toolsbeta]
00:31 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P89787 and previous config saved to /var/cache/conftool/dbconfig/20260304-003107-marostegui.json [production]
00:27 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster [toolsbeta]
00:25 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) [toolsbeta]
00:16 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P89786 and previous config saved to /var/cache/conftool/dbconfig/20260304-001559-marostegui.json [production]
00:09 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node [toolsbeta]
00:04 <root@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) [toolsbeta]
00:00 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1262 (T418465)', diff saved to https://phabricator.wikimedia.org/P89785 and previous config saved to /var/cache/conftool/dbconfig/20260304-000052-marostegui.json [production]
2026-03-03 §
23:56 <mutante> nevermind - puppet restarts it (debug) (T417458) [codesearch]
23:55 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
23:54 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
23:53 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
23:52 <James_F> Zuul: [mediawiki/extensions/WikimediaMessages] Drop MetricsPlatform phan dep [releng]
23:52 <James_F> Zuul: [mediawiki/extensions/WikimediaEvents] Drop MetricsPlatform phan dep [releng]
23:51 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
23:51 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]