|
2024-02-20
ยง
|
| 16:00 |
<bking@cumin2002> |
START - Cookbook sre.elasticsearch.ban Banning hosts: elastic2089*,elastic2062*,elastic2061* for switch maintenance - bking@cumin2002 - T355860 |
[production] |
| 15:59 |
<ayounsi@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2005.codfw.wmnet with reason: host reimage |
[production] |
| 15:58 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P57366 and previous config saved to /var/cache/conftool/dbconfig/20240220-155820-marostegui.json |
[production] |
| 15:57 |
<xcollazo> |
deployed latest Airflow DAG updates for the analytics instance |
[analytics] |
| 15:55 |
<xcollazo@deploy2002> |
Finished deploy [airflow-dags/analytics@b115452]: (no justification provided) (duration: 00m 34s) |
[production] |
| 15:55 |
<Emperor> |
import ceph-reef packages to apt1001 T279621 |
[production] |
| 15:55 |
<xcollazo@deploy2002> |
Started deploy [airflow-dags/analytics@b115452]: (no justification provided) |
[production] |
| 15:54 |
<dani@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/miscweb: apply |
[production] |
| 15:53 |
<dani@deploy2002> |
helmfile [codfw] START helmfile.d/services/miscweb: apply |
[production] |
| 15:53 |
<dani@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/miscweb: apply |
[production] |
| 15:50 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-101 |
[tools] |
| 15:50 |
<dani@deploy2002> |
helmfile [eqiad] START helmfile.d/services/miscweb: apply |
[production] |
| 15:50 |
<dani@deploy2002> |
helmfile [staging] DONE helmfile.d/services/miscweb: apply |
[production] |
| 15:50 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-101 |
[tools] |
| 15:49 |
<dani@deploy2002> |
helmfile [staging] START helmfile.d/services/miscweb: apply |
[production] |
| 15:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1233 (re)pooling @ 20%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57365 and previous config saved to /var/cache/conftool/dbconfig/20240220-154924-arnaudb.json |
[production] |
| 15:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1210 (re)pooling @ 20%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57364 and previous config saved to /var/cache/conftool/dbconfig/20240220-154920-arnaudb.json |
[production] |
| 15:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 20%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57363 and previous config saved to /var/cache/conftool/dbconfig/20240220-154920-arnaudb.json |
[production] |
| 15:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P57362 and previous config saved to /var/cache/conftool/dbconfig/20240220-154917-arnaudb.json |
[production] |
| 15:49 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster |
[tools] |
| 15:48 |
<taavi@cloudcumin1001> |
Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster |
[tools] |
| 15:46 |
<denisse> |
When doing the alert hosts upgrade we encountered some issues that prevented us to properly reimage the hosts to proceed with the upgrade. We're investigating this issue and inform of the new alert hosts upgrade date ASAP. - T333615 |
[production] |
| 15:46 |
<denisse> |
When doing the alert hosts upgrade we encountered some issues that prevented us to properly reimage the hosts to proceed with the upgrade. We're investigating this issue and inform of the new alert hosts upgrade date ASAP. - T333615 |
[production] |
| 15:46 |
<godog> |
re-enable meta-monitoring on wikitech-static.w.o - T333615 |
[production] |
| 15:43 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P57361 and previous config saved to /var/cache/conftool/dbconfig/20240220-154313-marostegui.json |
[production] |
| 15:42 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1233.eqiad.wmnet |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1168.eqiad.wmnet |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1226.eqiad.wmnet |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1210.eqiad.wmnet |
[production] |
| 15:40 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster |
[tools] |
| 15:39 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-102 |
[tools] |
| 15:39 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-102 |
[tools] |
| 15:38 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster |
[tools] |
| 15:38 |
<taavi@cloudcumin1001> |
Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster |
[tools] |
| 15:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1233.eqiad.wmnet |
[production] |
| 15:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1226.eqiad.wmnet |
[production] |
| 15:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1210.eqiad.wmnet |
[production] |
| 15:36 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1168.eqiad.wmnet |
[production] |
| 15:35 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1168 db1210 db1226 db1233 depool for T356240', diff saved to https://phabricator.wikimedia.org/P57359 and previous config saved to /var/cache/conftool/dbconfig/20240220-153557-arnaudb.json |
[production] |
| 15:34 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2128 (T357189)', diff saved to https://phabricator.wikimedia.org/P57358 and previous config saved to /var/cache/conftool/dbconfig/20240220-153410-arnaudb.json |
[production] |
| 15:33 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db[1168,1210,1226,1233].eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 15:33 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on db[1168,1210,1226,1233].eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 15:32 |
<godog> |
temp disable meta-monitoring on wikitech-static.w.o - T333615 |
[production] |
| 15:30 |
<Emperor> |
import ceph-reef packages to apt1001 T279621 |
[production] |
| 15:30 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2128 (T357189)', diff saved to https://phabricator.wikimedia.org/P57357 and previous config saved to /var/cache/conftool/dbconfig/20240220-153000-arnaudb.json |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2128.codfw.wmnet with reason: Maintenance |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2128.codfw.wmnet with reason: Maintenance |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2123 (T357189)', diff saved to https://phabricator.wikimedia.org/P57356 and previous config saved to /var/cache/conftool/dbconfig/20240220-152933-arnaudb.json |
[production] |