|
2026-01-09
ยง
|
| 10:49 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2223 (T413525)', diff saved to https://phabricator.wikimedia.org/P86939 and previous config saved to /var/cache/conftool/dbconfig/20260109-104942-marostegui.json |
[production] |
| 10:39 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P86938 and previous config saved to /var/cache/conftool/dbconfig/20260109-103934-marostegui.json |
[production] |
| 10:38 |
<gehel> |
depooling / repooling wdqs-main@eqiad servers one by one to allow time to recover and catch up on updates. |
[production] |
| 10:36 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.newpool (exit_code=0) pool es1049: test |
[production] |
| 10:29 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P86936 and previous config saved to /var/cache/conftool/dbconfig/20260109-102925-marostegui.json |
[production] |
| 10:19 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2223 (T413525)', diff saved to https://phabricator.wikimedia.org/P86934 and previous config saved to /var/cache/conftool/dbconfig/20260109-101917-marostegui.json |
[production] |
| 10:16 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2223 (T413525)', diff saved to https://phabricator.wikimedia.org/P86933 and previous config saved to /var/cache/conftool/dbconfig/20260109-101648-marostegui.json |
[production] |
| 10:16 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2223.codfw.wmnet with reason: Maintenance |
[production] |
| 10:16 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2213 (T413525)', diff saved to https://phabricator.wikimedia.org/P86932 and previous config saved to /var/cache/conftool/dbconfig/20260109-101622-marostegui.json |
[production] |
| 10:06 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P86930 and previous config saved to /var/cache/conftool/dbconfig/20260109-100614-marostegui.json |
[production] |
| 09:56 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P86929 and previous config saved to /var/cache/conftool/dbconfig/20260109-095606-marostegui.json |
[production] |
| 09:51 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.newpool pool es1049: test |
[production] |
| 09:50 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.newdepool (exit_code=0) depool es1049: test |
[production] |
| 09:50 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.newdepool depool es1049: test |
[production] |
| 09:45 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2213 (T413525)', diff saved to https://phabricator.wikimedia.org/P86926 and previous config saved to /var/cache/conftool/dbconfig/20260109-094558-marostegui.json |
[production] |
| 09:31 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2213 (T413525)', diff saved to https://phabricator.wikimedia.org/P86925 and previous config saved to /var/cache/conftool/dbconfig/20260109-093138-marostegui.json |
[production] |
| 09:31 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2213.codfw.wmnet with reason: Maintenance |
[production] |
| 09:31 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2211 (T413525)', diff saved to https://phabricator.wikimedia.org/P86924 and previous config saved to /var/cache/conftool/dbconfig/20260109-093114-marostegui.json |
[production] |
| 09:21 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P86923 and previous config saved to /var/cache/conftool/dbconfig/20260109-092105-marostegui.json |
[production] |
| 09:10 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P86922 and previous config saved to /var/cache/conftool/dbconfig/20260109-091058-marostegui.json |
[production] |
| 09:07 |
<urbanecm> |
[urbanecm@deploy2002 ~]$ kubectl delete job/growthexperiments-updatementeedata-s1-29460615 # T414167 |
[production] |
| 09:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2211 (T413525)', diff saved to https://phabricator.wikimedia.org/P86921 and previous config saved to /var/cache/conftool/dbconfig/20260109-090050-marostegui.json |
[production] |
| 08:48 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2211 (T413525)', diff saved to https://phabricator.wikimedia.org/P86920 and previous config saved to /var/cache/conftool/dbconfig/20260109-084834-marostegui.json |
[production] |
| 08:48 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2211.codfw.wmnet with reason: Maintenance |
[production] |
| 08:47 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2201.codfw.wmnet with reason: Maintenance |
[production] |
| 08:47 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2178 (T413525)', diff saved to https://phabricator.wikimedia.org/P86919 and previous config saved to /var/cache/conftool/dbconfig/20260109-084733-marostegui.json |
[production] |
| 08:37 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P86918 and previous config saved to /var/cache/conftool/dbconfig/20260109-083725-marostegui.json |
[production] |
| 08:30 |
<gehel> |
restarting blazegraph on wdqs-main@eqiad - high thread count |
[production] |
| 08:27 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P86917 and previous config saved to /var/cache/conftool/dbconfig/20260109-082717-marostegui.json |
[production] |
| 08:17 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2178 (T413525)', diff saved to https://phabricator.wikimedia.org/P86916 and previous config saved to /var/cache/conftool/dbconfig/20260109-081708-marostegui.json |
[production] |
| 08:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2178 (T413525)', diff saved to https://phabricator.wikimedia.org/P86915 and previous config saved to /var/cache/conftool/dbconfig/20260109-080058-marostegui.json |
[production] |
| 08:00 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2178.codfw.wmnet with reason: Maintenance |
[production] |
| 08:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2171 (T413525)', diff saved to https://phabricator.wikimedia.org/P86914 and previous config saved to /var/cache/conftool/dbconfig/20260109-080033-marostegui.json |
[production] |
| 07:50 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P86913 and previous config saved to /var/cache/conftool/dbconfig/20260109-075025-marostegui.json |
[production] |
| 07:40 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P86912 and previous config saved to /var/cache/conftool/dbconfig/20260109-074017-marostegui.json |
[production] |
| 07:30 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2171 (T413525)', diff saved to https://phabricator.wikimedia.org/P86911 and previous config saved to /var/cache/conftool/dbconfig/20260109-073008-marostegui.json |
[production] |
| 07:16 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2171 (T413525)', diff saved to https://phabricator.wikimedia.org/P86910 and previous config saved to /var/cache/conftool/dbconfig/20260109-071621-marostegui.json |
[production] |
| 07:16 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2171.codfw.wmnet with reason: Maintenance |
[production] |
| 07:16 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2157 (T413525)', diff saved to https://phabricator.wikimedia.org/P86909 and previous config saved to /var/cache/conftool/dbconfig/20260109-071606-marostegui.json |
[production] |
| 07:05 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P86908 and previous config saved to /var/cache/conftool/dbconfig/20260109-070558-marostegui.json |
[production] |
| 06:55 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P86907 and previous config saved to /var/cache/conftool/dbconfig/20260109-065549-marostegui.json |
[production] |
| 06:45 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2157 (T413525)', diff saved to https://phabricator.wikimedia.org/P86906 and previous config saved to /var/cache/conftool/dbconfig/20260109-064541-marostegui.json |
[production] |
| 06:31 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2157 (T413525)', diff saved to https://phabricator.wikimedia.org/P86905 and previous config saved to /var/cache/conftool/dbconfig/20260109-063154-marostegui.json |
[production] |
| 06:31 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2157.codfw.wmnet with reason: Maintenance |
[production] |
| 06:18 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1210.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:32 |
<eileen> |
civicrm upgraded from 9978d78e to 197629c6 |
[production] |
| 04:59 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2172 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86904 and previous config saved to /var/cache/conftool/dbconfig/20260109-045935-marostegui.json |
[production] |
| 04:59 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance |
[production] |
| 04:59 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2155 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86903 and previous config saved to /var/cache/conftool/dbconfig/20260109-045910-marostegui.json |
[production] |
| 04:49 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P86902 and previous config saved to /var/cache/conftool/dbconfig/20260109-044902-marostegui.json |
[production] |