|
2024-02-20
ยง
|
| 16:34 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1226 (re)pooling @ 100%: maintenance done', diff saved to https://phabricator.wikimedia.org/P57381 and previous config saved to /var/cache/conftool/dbconfig/20240220-163442-arnaudb.json |
[production] |
| 16:30 |
<fnegri@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1001.eqiad.wmnet |
[production] |
| 16:29 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bookworm |
[production] |
| 16:27 |
<sukhe@cumin2002> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4052.ulsfo.wmnet with OS bookworm |
[production] |
| 16:24 |
<fnegri@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host cloudrabbit1001.eqiad.wmnet |
[production] |
| 16:24 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2157 (T357189)', diff saved to https://phabricator.wikimedia.org/P57380 and previous config saved to /var/cache/conftool/dbconfig/20240220-162408-arnaudb.json |
[production] |
| 16:21 |
<fnegri@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1002.eqiad.wmnet |
[production] |
| 16:20 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2157 (T357189)', diff saved to https://phabricator.wikimedia.org/P57379 and previous config saved to /var/cache/conftool/dbconfig/20240220-161953-arnaudb.json |
[production] |
| 16:20 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance |
[production] |
| 16:20 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1233 (re)pooling @ 75%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57378 and previous config saved to /var/cache/conftool/dbconfig/20240220-161946-arnaudb.json |
[production] |
| 16:20 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1210 (re)pooling @ 75%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57377 and previous config saved to /var/cache/conftool/dbconfig/20240220-161942-arnaudb.json |
[production] |
| 16:19 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 75%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57376 and previous config saved to /var/cache/conftool/dbconfig/20240220-161942-arnaudb.json |
[production] |
| 16:19 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1226 (re)pooling @ 75%: maintenance done', diff saved to https://phabricator.wikimedia.org/P57375 and previous config saved to /var/cache/conftool/dbconfig/20240220-161937-arnaudb.json |
[production] |
| 16:19 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2157.codfw.wmnet with reason: Maintenance |
[production] |
| 16:19 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2128 (T357189)', diff saved to https://phabricator.wikimedia.org/P57374 and previous config saved to /var/cache/conftool/dbconfig/20240220-161931-arnaudb.json |
[production] |
| 16:18 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bookworm |
[production] |
| 16:14 |
<fnegri@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host cloudrabbit1002.eqiad.wmnet |
[production] |
| 16:13 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1218 (T355609)', diff saved to https://phabricator.wikimedia.org/P57373 and previous config saved to /var/cache/conftool/dbconfig/20240220-161348-marostegui.json |
[production] |
| 16:13 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1218.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:13 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1218.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:13 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1207 (T355609)', diff saved to https://phabricator.wikimedia.org/P57372 and previous config saved to /var/cache/conftool/dbconfig/20240220-161326-marostegui.json |
[production] |
| 16:12 |
<fnegri@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1003.eqiad.wmnet |
[production] |
| 16:11 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Unbanning all hosts in search_codfw |
[production] |
| 16:11 |
<bking@cumin2002> |
START - Cookbook sre.elasticsearch.ban Unbanning all hosts in search_codfw |
[production] |
| 16:09 |
<hnowlan@cumin2002> |
conftool action : set/weight=10:pooled=yes; selector: name=(mw2312.codfw.wmnet|mw2313.codfw.wmnet|mw2367.codfw.wmnet|mw2369.codfw.wmnet) |
[production] |
| 16:07 |
<topranks> |
Commencing network maintenance migrating servers to new switch codfw rack A7 T355867 |
[production] |
| 16:06 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 22 hosts with reason: Migrating servers in codfw rack A7 to lsw1-a7-codfw |
[production] |
| 16:06 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on 22 hosts with reason: Migrating servers in codfw rack A7 to lsw1-a7-codfw |
[production] |
| 16:05 |
<fnegri@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host cloudrabbit1003.eqiad.wmnet |
[production] |
| 16:05 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1210 (re)pooling @ 50%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57371 and previous config saved to /var/cache/conftool/dbconfig/20240220-160438-arnaudb.json |
[production] |
| 16:05 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1168 (re)pooling @ 50%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57370 and previous config saved to /var/cache/conftool/dbconfig/20240220-160437-arnaudb.json |
[production] |
| 16:04 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1226 (re)pooling @ 50%: maintenance done', diff saved to https://phabricator.wikimedia.org/P57369 and previous config saved to /var/cache/conftool/dbconfig/20240220-160432-arnaudb.json |
[production] |
| 16:04 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db1233 (re)pooling @ 50%: Maintenance done', diff saved to https://phabricator.wikimedia.org/P57368 and previous config saved to /var/cache/conftool/dbconfig/20240220-160429-arnaudb.json |
[production] |
| 16:04 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P57367 and previous config saved to /var/cache/conftool/dbconfig/20240220-160423-arnaudb.json |
[production] |
| 16:02 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on asw-a-codfw,cr[1-2]-codfw,lsw1-a7-codfw.mgmt with reason: prepping for server uplink migration codfw rack a7 |
[production] |
| 16:02 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on asw-a-codfw,cr[1-2]-codfw,lsw1-a7-codfw.mgmt with reason: prepping for server uplink migration codfw rack a7 |
[production] |
| 16:02 |
<ayounsi@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2005.codfw.wmnet with reason: host reimage |
[production] |
| 16:00 |
<hnowlan> |
running `homer 'cr*codfw*' commit 'T351074'` for new k8s workers |
[production] |
| 16:00 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic2089*,elastic2062*,elastic2061* for switch maintenance - bking@cumin2002 - T355860 |
[production] |
| 16:00 |
<bking@cumin2002> |
START - Cookbook sre.elasticsearch.ban Banning hosts: elastic2089*,elastic2062*,elastic2061* for switch maintenance - bking@cumin2002 - T355860 |
[production] |
| 15:59 |
<ayounsi@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2005.codfw.wmnet with reason: host reimage |
[production] |
| 15:58 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P57366 and previous config saved to /var/cache/conftool/dbconfig/20240220-155820-marostegui.json |
[production] |
| 15:55 |
<xcollazo@deploy2002> |
Finished deploy [airflow-dags/analytics@b115452]: (no justification provided) (duration: 00m 34s) |
[production] |
| 15:55 |
<Emperor> |
import ceph-reef packages to apt1001 T279621 |
[production] |
| 15:55 |
<xcollazo@deploy2002> |
Started deploy [airflow-dags/analytics@b115452]: (no justification provided) |
[production] |
| 15:54 |
<dani@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/miscweb: apply |
[production] |
| 15:53 |
<dani@deploy2002> |
helmfile [codfw] START helmfile.d/services/miscweb: apply |
[production] |
| 15:53 |
<dani@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/miscweb: apply |
[production] |
| 15:50 |
<dani@deploy2002> |
helmfile [eqiad] START helmfile.d/services/miscweb: apply |
[production] |
| 15:50 |
<dani@deploy2002> |
helmfile [staging] DONE helmfile.d/services/miscweb: apply |
[production] |