|
2024-02-20
ยง
|
| 15:53 |
<dani@deploy2002> |
helmfile [codfw] START helmfile.d/services/miscweb: apply |
[production] |
| 15:53 |
<dani@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/miscweb: apply |
[production] |
| 15:50 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-101 |
[tools] |
| 15:50 |
<dani@deploy2002> |
helmfile [eqiad] START helmfile.d/services/miscweb: apply |
[production] |
| 15:50 |
<dani@deploy2002> |
helmfile [staging] DONE helmfile.d/services/miscweb: apply |
[production] |
| 15:50 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-101 |
[tools] |
| 15:49 |
<dani@deploy2002> |
helmfile [staging] START helmfile.d/services/miscweb: apply |
[production] |
| 15:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1233 (re)pooling @ 20%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57365 and previous config saved to /var/cache/conftool/dbconfig/20240220-154924-arnaudb.json |
[production] |
| 15:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1210 (re)pooling @ 20%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57364 and previous config saved to /var/cache/conftool/dbconfig/20240220-154920-arnaudb.json |
[production] |
| 15:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 20%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57363 and previous config saved to /var/cache/conftool/dbconfig/20240220-154920-arnaudb.json |
[production] |
| 15:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P57362 and previous config saved to /var/cache/conftool/dbconfig/20240220-154917-arnaudb.json |
[production] |
| 15:49 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster |
[tools] |
| 15:48 |
<taavi@cloudcumin1001> |
Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster |
[tools] |
| 15:46 |
<denisse> |
When doing the alert hosts upgrade we encountered some issues that prevented us to properly reimage the hosts to proceed with the upgrade. We're investigating this issue and inform of the new alert hosts upgrade date ASAP. - T333615 |
[production] |
| 15:46 |
<denisse> |
When doing the alert hosts upgrade we encountered some issues that prevented us to properly reimage the hosts to proceed with the upgrade. We're investigating this issue and inform of the new alert hosts upgrade date ASAP. - T333615 |
[production] |
| 15:46 |
<godog> |
re-enable meta-monitoring on wikitech-static.w.o - T333615 |
[production] |
| 15:43 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P57361 and previous config saved to /var/cache/conftool/dbconfig/20240220-154313-marostegui.json |
[production] |
| 15:42 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1233.eqiad.wmnet |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1168.eqiad.wmnet |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1226.eqiad.wmnet |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1210.eqiad.wmnet |
[production] |
| 15:40 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster |
[tools] |
| 15:39 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-102 |
[tools] |
| 15:39 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-102 |
[tools] |
| 15:38 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster |
[tools] |
| 15:38 |
<taavi@cloudcumin1001> |
Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster |
[tools] |
| 15:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1233.eqiad.wmnet |
[production] |
| 15:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1226.eqiad.wmnet |
[production] |
| 15:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1210.eqiad.wmnet |
[production] |
| 15:36 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1168.eqiad.wmnet |
[production] |
| 15:35 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1168 db1210 db1226 db1233 depool for T356240', diff saved to https://phabricator.wikimedia.org/P57359 and previous config saved to /var/cache/conftool/dbconfig/20240220-153557-arnaudb.json |
[production] |
| 15:34 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2128 (T357189)', diff saved to https://phabricator.wikimedia.org/P57358 and previous config saved to /var/cache/conftool/dbconfig/20240220-153410-arnaudb.json |
[production] |
| 15:33 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db[1168,1210,1226,1233].eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 15:33 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on db[1168,1210,1226,1233].eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 15:32 |
<godog> |
temp disable meta-monitoring on wikitech-static.w.o - T333615 |
[production] |
| 15:30 |
<Emperor> |
import ceph-reef packages to apt1001 T279621 |
[production] |
| 15:30 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2128 (T357189)', diff saved to https://phabricator.wikimedia.org/P57357 and previous config saved to /var/cache/conftool/dbconfig/20240220-153000-arnaudb.json |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2128.codfw.wmnet with reason: Maintenance |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2128.codfw.wmnet with reason: Maintenance |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2123 (T357189)', diff saved to https://phabricator.wikimedia.org/P57356 and previous config saved to /var/cache/conftool/dbconfig/20240220-152933-arnaudb.json |
[production] |
| 15:29 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster |
[tools] |
| 15:28 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1207 (T355609)', diff saved to https://phabricator.wikimedia.org/P57355 and previous config saved to /var/cache/conftool/dbconfig/20240220-152807-marostegui.json |
[production] |
| 15:25 |
<ayounsi@cumin1002> |
START - Cookbook sre.hosts.reimage for host sretest2005.codfw.wmnet with OS bookworm |
[production] |
| 15:23 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud |
[tools] |
| 15:21 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud |
[tools] |
| 15:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2170 (re)pooling @ 100%: After migration', diff saved to https://phabricator.wikimedia.org/P57354 and previous config saved to /var/cache/conftool/dbconfig/20240220-151812-root.json |
[production] |
| 15:16 |
<dcausse> |
depooled wdqs2009 & wdqs2020 (T355867) |
[production] |
| 15:16 |
<denisse_> |
starting the Alert hosts upgrade to Bookworm - T333615 |
[production] |