7701-7750 of 10000 results (46ms)
2024-02-20 ยง
12:38 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1213', diff saved to https://phabricator.wikimedia.org/P57309 and previous config saved to /var/cache/conftool/dbconfig/20240220-123804-arnaudb.json [production]
12:30 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster [tools]
12:30 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-99 [tools]
12:29 <marostegui@cumin1002> dbctl commit (dc=all): 'Depooling db1196 (T355609)', diff saved to https://phabricator.wikimedia.org/P57308 and previous config saved to /var/cache/conftool/dbconfig/20240220-122947-marostegui.json [production]
12:29 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
12:29 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-99 [tools]
12:29 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
12:29 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1196.eqiad.wmnet with reason: Maintenance [production]
12:29 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster [tools]
12:29 <taavi@cloudcumin1001> Added a new k8s worker-nfs tools-k8s-worker-nfs-54.tools.eqiad1.wikimedia.cloud to the cluster [tools]
12:29 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 6:00:00 on db1196.eqiad.wmnet with reason: Maintenance [production]
12:29 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186 (T355609)', diff saved to https://phabricator.wikimedia.org/P57307 and previous config saved to /var/cache/conftool/dbconfig/20240220-122907-marostegui.json [production]
12:22 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1213', diff saved to https://phabricator.wikimedia.org/P57306 and previous config saved to /var/cache/conftool/dbconfig/20240220-122258-arnaudb.json [production]
12:20 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster [tools]
12:19 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-98 [tools]
12:19 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-98 [tools]
12:18 <hnowlan@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mw2384.codfw.wmnet with OS bullseye [production]
12:18 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster [tools]
12:18 <taavi@cloudcumin1001> Added a new k8s worker-nfs tools-k8s-worker-nfs-53.tools.eqiad1.wikimedia.cloud to the cluster [tools]
12:18 <hnowlan@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mw2385.codfw.wmnet with OS bullseye [production]
12:16 <claime> Draining mw2379 [production]
12:14 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P57305 and previous config saved to /var/cache/conftool/dbconfig/20240220-121402-marostegui.json [production]
12:09 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster [tools]
12:07 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1213 (T357189)', diff saved to https://phabricator.wikimedia.org/P57304 and previous config saved to /var/cache/conftool/dbconfig/20240220-120752-arnaudb.json [production]
12:06 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-97 [tools]
12:05 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-97 [tools]
12:05 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1213 (T357189)', diff saved to https://phabricator.wikimedia.org/P57303 and previous config saved to /var/cache/conftool/dbconfig/20240220-120434-arnaudb.json [production]
12:05 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1213.eqiad.wmnet with reason: Maintenance [production]
12:04 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1213.eqiad.wmnet with reason: Maintenance [production]
12:04 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1210 (T357189)', diff saved to https://phabricator.wikimedia.org/P57302 and previous config saved to /var/cache/conftool/dbconfig/20240220-120412-arnaudb.json [production]
12:04 <kart_> cxserver: Update to 2024-02-15-085232-production + Bump mesh.configuration to 1.7 (T333969, T352747, T355686, T255568) [production]
12:03 <hnowlan@cumin1002> START - Cookbook sre.hosts.reimage for host mw2385.codfw.wmnet with OS bullseye [production]
12:03 <hnowlan@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mw2385.codfw.wmnet with OS bullseye [production]
12:02 <hnowlan@cumin1002> START - Cookbook sre.hosts.reimage for host mw2384.codfw.wmnet with OS bullseye [production]
12:02 <hnowlan@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mw2384.codfw.wmnet with OS bullseye [production]
12:01 <hnowlan@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2369.codfw.wmnet with OS bullseye [production]
12:00 <marostegui@cumin1002> dbctl commit (dc=all): 'db2169 (re)pooling @ 100%: After rearraging sections T354826', diff saved to https://phabricator.wikimedia.org/P57301 and previous config saved to /var/cache/conftool/dbconfig/20240220-120031-root.json [production]
12:00 <kartik@deploy2002> helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [production]
11:59 <kartik@deploy2002> helmfile [eqiad] START helmfile.d/services/cxserver: apply [production]
11:58 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P57300 and previous config saved to /var/cache/conftool/dbconfig/20240220-115855-marostegui.json [production]
11:57 <hnowlan@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2367.codfw.wmnet with OS bullseye [production]
11:56 <wmbot~taavi@runko> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 [toolsbeta]
11:56 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster [tools]
11:56 <taavi@cloudcumin1001> Added a new k8s worker-nfs tools-k8s-worker-nfs-52.tools.eqiad1.wikimedia.cloud to the cluster [tools]
11:56 <wmbot~taavi@runko> START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1 [toolsbeta]
11:56 <wmbot~taavi@runko> END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 [toolsbeta]
11:55 <wmbot~taavi@runko> START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1 [toolsbeta]
11:55 <hnowlan@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2313.codfw.wmnet with OS bullseye [production]
11:55 <kartik@deploy2002> helmfile [codfw] DONE helmfile.d/services/cxserver: apply [production]
11:54 <kartik@deploy2002> helmfile [codfw] START helmfile.d/services/cxserver: apply [production]