|
2026-01-09
ยง
|
| 11:23 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2228 (T413525)', diff saved to https://phabricator.wikimedia.org/P86944 and previous config saved to /var/cache/conftool/dbconfig/20260109-112302-marostegui.json |
[production] |
| 11:15 |
<jelto@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/miscweb: apply |
[production] |
| 11:14 |
<jelto@deploy2002> |
helmfile [eqiad] START helmfile.d/services/miscweb: apply |
[production] |
| 11:14 |
<jelto@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/miscweb: apply |
[production] |
| 11:14 |
<jelto@deploy2002> |
helmfile [codfw] START helmfile.d/services/miscweb: apply |
[production] |
| 11:14 |
<jelto@deploy2002> |
helmfile [staging] DONE helmfile.d/services/miscweb: apply |
[production] |
| 11:13 |
<jelto@deploy2002> |
helmfile [staging] START helmfile.d/services/miscweb: apply |
[production] |
| 11:12 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P86943 and previous config saved to /var/cache/conftool/dbconfig/20260109-111254-marostegui.json |
[production] |
| 11:09 |
<moritzm> |
revoked legacy config-master discovery cert T365798 |
[production] |
| 11:02 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P86942 and previous config saved to /var/cache/conftool/dbconfig/20260109-110245-marostegui.json |
[production] |
| 10:52 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2228 (T413525)', diff saved to https://phabricator.wikimedia.org/P86941 and previous config saved to /var/cache/conftool/dbconfig/20260109-105237-marostegui.json |
[production] |
| 10:50 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2228 (T413525)', diff saved to https://phabricator.wikimedia.org/P86940 and previous config saved to /var/cache/conftool/dbconfig/20260109-105008-marostegui.json |
[production] |
| 10:50 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2228.codfw.wmnet with reason: Maintenance |
[production] |
| 10:49 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2223 (T413525)', diff saved to https://phabricator.wikimedia.org/P86939 and previous config saved to /var/cache/conftool/dbconfig/20260109-104942-marostegui.json |
[production] |
| 10:39 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P86938 and previous config saved to /var/cache/conftool/dbconfig/20260109-103934-marostegui.json |
[production] |
| 10:38 |
<gehel> |
depooling / repooling wdqs-main@eqiad servers one by one to allow time to recover and catch up on updates. |
[production] |
| 10:36 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.newpool (exit_code=0) pool es1049: test |
[production] |
| 10:29 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P86936 and previous config saved to /var/cache/conftool/dbconfig/20260109-102925-marostegui.json |
[production] |
| 10:19 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2223 (T413525)', diff saved to https://phabricator.wikimedia.org/P86934 and previous config saved to /var/cache/conftool/dbconfig/20260109-101917-marostegui.json |
[production] |
| 10:16 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2223 (T413525)', diff saved to https://phabricator.wikimedia.org/P86933 and previous config saved to /var/cache/conftool/dbconfig/20260109-101648-marostegui.json |
[production] |
| 10:16 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2223.codfw.wmnet with reason: Maintenance |
[production] |
| 10:16 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2213 (T413525)', diff saved to https://phabricator.wikimedia.org/P86932 and previous config saved to /var/cache/conftool/dbconfig/20260109-101622-marostegui.json |
[production] |
| 10:06 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P86930 and previous config saved to /var/cache/conftool/dbconfig/20260109-100614-marostegui.json |
[production] |
| 09:56 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P86929 and previous config saved to /var/cache/conftool/dbconfig/20260109-095606-marostegui.json |
[production] |
| 09:51 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.newpool pool es1049: test |
[production] |
| 09:50 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.newdepool (exit_code=0) depool es1049: test |
[production] |
| 09:50 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.newdepool depool es1049: test |
[production] |
| 09:45 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2213 (T413525)', diff saved to https://phabricator.wikimedia.org/P86926 and previous config saved to /var/cache/conftool/dbconfig/20260109-094558-marostegui.json |
[production] |
| 09:31 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2213 (T413525)', diff saved to https://phabricator.wikimedia.org/P86925 and previous config saved to /var/cache/conftool/dbconfig/20260109-093138-marostegui.json |
[production] |
| 09:31 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2213.codfw.wmnet with reason: Maintenance |
[production] |
| 09:31 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2211 (T413525)', diff saved to https://phabricator.wikimedia.org/P86924 and previous config saved to /var/cache/conftool/dbconfig/20260109-093114-marostegui.json |
[production] |
| 09:21 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P86923 and previous config saved to /var/cache/conftool/dbconfig/20260109-092105-marostegui.json |
[production] |
| 09:10 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P86922 and previous config saved to /var/cache/conftool/dbconfig/20260109-091058-marostegui.json |
[production] |
| 09:07 |
<urbanecm> |
[urbanecm@deploy2002 ~]$ kubectl delete job/growthexperiments-updatementeedata-s1-29460615 # T414167 |
[production] |
| 09:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2211 (T413525)', diff saved to https://phabricator.wikimedia.org/P86921 and previous config saved to /var/cache/conftool/dbconfig/20260109-090050-marostegui.json |
[production] |
| 08:48 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2211 (T413525)', diff saved to https://phabricator.wikimedia.org/P86920 and previous config saved to /var/cache/conftool/dbconfig/20260109-084834-marostegui.json |
[production] |
| 08:48 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2211.codfw.wmnet with reason: Maintenance |
[production] |
| 08:47 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2201.codfw.wmnet with reason: Maintenance |
[production] |
| 08:47 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2178 (T413525)', diff saved to https://phabricator.wikimedia.org/P86919 and previous config saved to /var/cache/conftool/dbconfig/20260109-084733-marostegui.json |
[production] |
| 08:37 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P86918 and previous config saved to /var/cache/conftool/dbconfig/20260109-083725-marostegui.json |
[production] |
| 08:30 |
<gehel> |
restarting blazegraph on wdqs-main@eqiad - high thread count |
[production] |
| 08:27 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P86917 and previous config saved to /var/cache/conftool/dbconfig/20260109-082717-marostegui.json |
[production] |
| 08:17 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2178 (T413525)', diff saved to https://phabricator.wikimedia.org/P86916 and previous config saved to /var/cache/conftool/dbconfig/20260109-081708-marostegui.json |
[production] |
| 08:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2178 (T413525)', diff saved to https://phabricator.wikimedia.org/P86915 and previous config saved to /var/cache/conftool/dbconfig/20260109-080058-marostegui.json |
[production] |
| 08:00 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2178.codfw.wmnet with reason: Maintenance |
[production] |
| 08:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2171 (T413525)', diff saved to https://phabricator.wikimedia.org/P86914 and previous config saved to /var/cache/conftool/dbconfig/20260109-080033-marostegui.json |
[production] |
| 07:50 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P86913 and previous config saved to /var/cache/conftool/dbconfig/20260109-075025-marostegui.json |
[production] |
| 07:40 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P86912 and previous config saved to /var/cache/conftool/dbconfig/20260109-074017-marostegui.json |
[production] |
| 07:30 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2171 (T413525)', diff saved to https://phabricator.wikimedia.org/P86911 and previous config saved to /var/cache/conftool/dbconfig/20260109-073008-marostegui.json |
[production] |
| 07:16 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2171 (T413525)', diff saved to https://phabricator.wikimedia.org/P86910 and previous config saved to /var/cache/conftool/dbconfig/20260109-071621-marostegui.json |
[production] |