|
2024-02-20
ยง
|
| 16:11 |
<bking@cumin2002> |
START - Cookbook sre.elasticsearch.ban Unbanning all hosts in search_codfw |
[production] |
| 16:09 |
<hnowlan@cumin2002> |
conftool action : set/weight=10:pooled=yes; selector: name=(mw2312.codfw.wmnet|mw2313.codfw.wmnet|mw2367.codfw.wmnet|mw2369.codfw.wmnet) |
[production] |
| 16:07 |
<topranks> |
Commencing network maintenance migrating servers to new switch codfw rack A7 T355867 |
[production] |
| 16:06 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 22 hosts with reason: Migrating servers in codfw rack A7 to lsw1-a7-codfw |
[production] |
| 16:06 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on 22 hosts with reason: Migrating servers in codfw rack A7 to lsw1-a7-codfw |
[production] |
| 16:05 |
<fnegri@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host cloudrabbit1003.eqiad.wmnet |
[production] |
| 16:05 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1210 (re)pooling @ 50%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57371 and previous config saved to /var/cache/conftool/dbconfig/20240220-160438-arnaudb.json |
[production] |
| 16:05 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 50%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57370 and previous config saved to /var/cache/conftool/dbconfig/20240220-160437-arnaudb.json |
[production] |
| 16:04 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1226 (re)pooling @ 50%: maintenance done', diff saved to https://phabricator.wikimedia.org/P57369 and previous config saved to /var/cache/conftool/dbconfig/20240220-160432-arnaudb.json |
[production] |
| 16:04 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1233 (re)pooling @ 50%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57368 and previous config saved to /var/cache/conftool/dbconfig/20240220-160429-arnaudb.json |
[production] |
| 16:04 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P57367 and previous config saved to /var/cache/conftool/dbconfig/20240220-160423-arnaudb.json |
[production] |
| 16:02 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on asw-a-codfw,cr[1-2]-codfw,lsw1-a7-codfw.mgmt with reason: prepping for server uplink migration codfw rack a7 |
[production] |
| 16:02 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on asw-a-codfw,cr[1-2]-codfw,lsw1-a7-codfw.mgmt with reason: prepping for server uplink migration codfw rack a7 |
[production] |
| 16:02 |
<ayounsi@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2005.codfw.wmnet with reason: host reimage |
[production] |
| 16:00 |
<hnowlan> |
running `homer 'cr*codfw*' commit 'T351074'` for new k8s workers |
[production] |
| 16:00 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic2089*,elastic2062*,elastic2061* for switch maintenance - bking@cumin2002 - T355860 |
[production] |
| 16:00 |
<bking@cumin2002> |
START - Cookbook sre.elasticsearch.ban Banning hosts: elastic2089*,elastic2062*,elastic2061* for switch maintenance - bking@cumin2002 - T355860 |
[production] |
| 15:59 |
<ayounsi@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2005.codfw.wmnet with reason: host reimage |
[production] |
| 15:58 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P57366 and previous config saved to /var/cache/conftool/dbconfig/20240220-155820-marostegui.json |
[production] |
| 15:55 |
<xcollazo@deploy2002> |
Finished deploy [airflow-dags/analytics@b115452]: (no justification provided) (duration: 00m 34s) |
[production] |
| 15:55 |
<Emperor> |
import ceph-reef packages to apt1001 T279621 |
[production] |
| 15:55 |
<xcollazo@deploy2002> |
Started deploy [airflow-dags/analytics@b115452]: (no justification provided) |
[production] |
| 15:54 |
<dani@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/miscweb: apply |
[production] |
| 15:53 |
<dani@deploy2002> |
helmfile [codfw] START helmfile.d/services/miscweb: apply |
[production] |
| 15:53 |
<dani@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/miscweb: apply |
[production] |
| 15:50 |
<dani@deploy2002> |
helmfile [eqiad] START helmfile.d/services/miscweb: apply |
[production] |
| 15:50 |
<dani@deploy2002> |
helmfile [staging] DONE helmfile.d/services/miscweb: apply |
[production] |
| 15:49 |
<dani@deploy2002> |
helmfile [staging] START helmfile.d/services/miscweb: apply |
[production] |
| 15:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1233 (re)pooling @ 20%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57365 and previous config saved to /var/cache/conftool/dbconfig/20240220-154924-arnaudb.json |
[production] |
| 15:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1210 (re)pooling @ 20%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57364 and previous config saved to /var/cache/conftool/dbconfig/20240220-154920-arnaudb.json |
[production] |
| 15:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 20%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57363 and previous config saved to /var/cache/conftool/dbconfig/20240220-154920-arnaudb.json |
[production] |
| 15:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P57362 and previous config saved to /var/cache/conftool/dbconfig/20240220-154917-arnaudb.json |
[production] |
| 15:46 |
<denisse> |
When doing the alert hosts upgrade we encountered some issues that prevented us to properly reimage the hosts to proceed with the upgrade. We're investigating this issue and inform of the new alert hosts upgrade date ASAP. - T333615 |
[production] |
| 15:46 |
<denisse> |
When doing the alert hosts upgrade we encountered some issues that prevented us to properly reimage the hosts to proceed with the upgrade. We're investigating this issue and inform of the new alert hosts upgrade date ASAP. - T333615 |
[production] |
| 15:46 |
<godog> |
re-enable meta-monitoring on wikitech-static.w.o - T333615 |
[production] |
| 15:43 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P57361 and previous config saved to /var/cache/conftool/dbconfig/20240220-154313-marostegui.json |
[production] |
| 15:42 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1233.eqiad.wmnet |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1168.eqiad.wmnet |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1226.eqiad.wmnet |
[production] |
| 15:41 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1210.eqiad.wmnet |
[production] |
| 15:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1233.eqiad.wmnet |
[production] |
| 15:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1226.eqiad.wmnet |
[production] |
| 15:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1210.eqiad.wmnet |
[production] |
| 15:36 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.upgrade for db1168.eqiad.wmnet |
[production] |
| 15:35 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1168 db1210 db1226 db1233 depool for T356240', diff saved to https://phabricator.wikimedia.org/P57359 and previous config saved to /var/cache/conftool/dbconfig/20240220-153557-arnaudb.json |
[production] |
| 15:34 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2128 (T357189)', diff saved to https://phabricator.wikimedia.org/P57358 and previous config saved to /var/cache/conftool/dbconfig/20240220-153410-arnaudb.json |
[production] |
| 15:33 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db[1168,1210,1226,1233].eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 15:33 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on db[1168,1210,1226,1233].eqiad.wmnet with reason: Silence for reboot T356240 |
[production] |
| 15:32 |
<godog> |
temp disable meta-monitoring on wikitech-static.w.o - T333615 |
[production] |
| 15:30 |
<Emperor> |
import ceph-reef packages to apt1001 T279621 |
[production] |