8401-8450 of 10000 results (31ms)
2026-03-04 ยง
05:43 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2179.codfw.wmnet with reason: Maintenance [production]
05:42 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1160.eqiad.wmnet with reason: Maintenance [production]
04:50 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
04:49 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
04:48 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
04:48 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
04:47 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
04:47 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
04:46 <root@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=0) [toolsbeta]
04:41 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster [toolsbeta]
04:37 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) [toolsbeta]
04:36 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster [toolsbeta]
04:32 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) [toolsbeta]
04:15 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_etcd_node [toolsbeta]
04:13 <root@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) [toolsbeta]
04:05 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
04:04 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
04:04 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
04:03 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
04:03 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
03:32 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
03:31 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
03:21 <ejegg> payments-wiki upgraded from 5e4939a3 to 9ae5bf60 [fundraising]
03:20 <ejegg> standalone (IPN listener) SmashPig upgraded from 78960c68 to 217fc7fc [fundraising]
03:09 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
03:08 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
03:06 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
03:05 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
02:50 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
02:49 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
02:47 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
02:43 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
02:37 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
02:36 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
02:08 <mwpresync@deploy2002> Finished scap build-images: Publishing wmf/next image (duration: 08m 04s) [production]
02:03 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
02:03 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
02:00 <mwpresync@deploy2002> Started scap build-images: Publishing wmf/next image [production]
01:57 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
01:57 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
01:57 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
01:56 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1263 (T418465)', diff saved to https://phabricator.wikimedia.org/P89793 and previous config saved to /var/cache/conftool/dbconfig/20260304-015657-marostegui.json [production]
01:53 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) [toolsbeta]
01:52 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_etcd_node [toolsbeta]
01:50 <root@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) [toolsbeta]
01:49 <root@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster [toolsbeta]
01:41 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P89792 and previous config saved to /var/cache/conftool/dbconfig/20260304-014150-marostegui.json [production]
01:26 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P89791 and previous config saved to /var/cache/conftool/dbconfig/20260304-012642-marostegui.json [production]
01:23 <zabe@deploy2002> helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply [production]
01:22 <zabe@deploy2002> helmfile [eqiad] START helmfile.d/services/mw-experimental: apply [production]