|
2024-02-20
ยง
|
| 16:02 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on asw-a-codfw,cr[1-2]-codfw,lsw1-a7-codfw.mgmt with reason: prepping for server uplink migration codfw rack a7 |
[production] |
| 16:02 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on asw-a-codfw,cr[1-2]-codfw,lsw1-a7-codfw.mgmt with reason: prepping for server uplink migration codfw rack a7 |
[production] |
| 16:02 |
<ayounsi@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2005.codfw.wmnet with reason: host reimage |
[production] |
| 16:00 |
<hnowlan> |
running `homer 'cr*codfw*' commit 'T351074'` for new k8s workers |
[production] |
| 16:00 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic2089*,elastic2062*,elastic2061* for switch maintenance - bking@cumin2002 - T355860 |
[production] |
| 16:00 |
<bking@cumin2002> |
START - Cookbook sre.elasticsearch.ban Banning hosts: elastic2089*,elastic2062*,elastic2061* for switch maintenance - bking@cumin2002 - T355860 |
[production] |
| 15:59 |
<ayounsi@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2005.codfw.wmnet with reason: host reimage |
[production] |
| 15:58 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P57366 and previous config saved to /var/cache/conftool/dbconfig/20240220-155820-marostegui.json |
[production] |
| 15:55 |
<xcollazo@deploy2002> |
Finished deploy [airflow-dags/analytics@b115452]: (no justification provided) (duration: 00m 34s) |
[production] |
| 15:55 |
<Emperor> |
import ceph-reef packages to apt1001 T279621 |
[production] |
| 15:55 |
<xcollazo@deploy2002> |
Started deploy [airflow-dags/analytics@b115452]: (no justification provided) |
[production] |
| 15:54 |
<dani@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/miscweb: apply |
[production] |
| 15:53 |
<dani@deploy2002> |
helmfile [codfw] START helmfile.d/services/miscweb: apply |
[production] |
| 15:53 |
<dani@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/miscweb: apply |
[production] |
| 15:50 |
<dani@deploy2002> |
helmfile [eqiad] START helmfile.d/services/miscweb: apply |
[production] |
| 15:50 |
<dani@deploy2002> |
helmfile [staging] DONE helmfile.d/services/miscweb: apply |
[production] |
| 15:49 |
<dani@deploy2002> |
helmfile [staging] START helmfile.d/services/miscweb: apply |
[production] |
| 15:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1233 (re)pooling @ 20%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57365 and previous config saved to /var/cache/conftool/dbconfig/20240220-154924-arnaudb.json |
[production] |
| 15:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1210 (re)pooling @ 20%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57364 and previous config saved to /var/cache/conftool/dbconfig/20240220-154920-arnaudb.json |
[production] |
| 15:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 20%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57363 and previous config saved to /var/cache/conftool/dbconfig/20240220-154920-arnaudb.json |
[production] |
| 15:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P57362 and previous config saved to /var/cache/conftool/dbconfig/20240220-154917-arnaudb.json |
[production] |
| 15:46 |
<denisse> |
When doing the alert hosts upgrade we encountered some issues that prevented us to properly reimage the hosts to proceed with the upgrade. We're investigating this issue and inform of the new alert hosts upgrade date ASAP. - T333615 |
[production] |
| 15:46 |
<denisse> |
When doing the alert hosts upgrade we encountered some issues that prevented us to properly reimage the hosts to proceed with the upgrade. We're investigating this issue and inform of the new alert hosts upgrade date ASAP. - T333615 |
[production] |
| 15:46 |
<godog> |
re-enable meta-monitoring on wikitech-static.w.o - T333615 |
[production] |
| 15:43 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P57361 and previous config saved to /var/cache/conftool/dbconfig/20240220-154313-marostegui.json |
[production] |
| 15:42 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1233.eqiad.wmnet |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1168.eqiad.wmnet |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1226.eqiad.wmnet |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1210.eqiad.wmnet |
[production] |
| 15:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1233.eqiad.wmnet |
[production] |
| 15:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1226.eqiad.wmnet |
[production] |
| 15:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1210.eqiad.wmnet |
[production] |
| 15:36 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1168.eqiad.wmnet |
[production] |
| 15:35 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1168 db1210 db1226 db1233 depool for T356240', diff saved to https://phabricator.wikimedia.org/P57359 and previous config saved to /var/cache/conftool/dbconfig/20240220-153557-arnaudb.json |
[production] |
| 15:34 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2128 (T357189)', diff saved to https://phabricator.wikimedia.org/P57358 and previous config saved to /var/cache/conftool/dbconfig/20240220-153410-arnaudb.json |
[production] |
| 15:33 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db[1168,1210,1226,1233].eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 15:33 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on db[1168,1210,1226,1233].eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 15:32 |
<godog> |
temp disable meta-monitoring on wikitech-static.w.o - T333615 |
[production] |
| 15:30 |
<Emperor> |
import ceph-reef packages to apt1001 T279621 |
[production] |
| 15:30 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2128 (T357189)', diff saved to https://phabricator.wikimedia.org/P57357 and previous config saved to /var/cache/conftool/dbconfig/20240220-153000-arnaudb.json |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2128.codfw.wmnet with reason: Maintenance |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2128.codfw.wmnet with reason: Maintenance |
[production] |
| 15:29 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2123 (T357189)', diff saved to https://phabricator.wikimedia.org/P57356 and previous config saved to /var/cache/conftool/dbconfig/20240220-152933-arnaudb.json |
[production] |
| 15:28 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1207 (T355609)', diff saved to https://phabricator.wikimedia.org/P57355 and previous config saved to /var/cache/conftool/dbconfig/20240220-152807-marostegui.json |
[production] |
| 15:25 |
<ayounsi@cumin1002> |
START - Cookbook sre.hosts.reimage for host sretest2005.codfw.wmnet with OS bookworm |
[production] |
| 15:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2170 (re)pooling @ 100%: After migration', diff saved to https://phabricator.wikimedia.org/P57354 and previous config saved to /var/cache/conftool/dbconfig/20240220-151812-root.json |
[production] |
| 15:16 |
<dcausse> |
depooled wdqs2009 & wdqs2020 (T355867) |
[production] |
| 15:16 |
<denisse_> |
starting the Alert hosts upgrade to Bookworm - T333615 |
[production] |