|
2026-02-27
ยง
|
| 09:21 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2193 (T418465)', diff saved to https://phabricator.wikimedia.org/P89104 and previous config saved to /var/cache/conftool/dbconfig/20260227-092101-marostegui.json |
[production] |
| 09:19 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy1027.eqiad.wmnet with OS trixie |
[production] |
| 09:18 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2193 (T418465)', diff saved to https://phabricator.wikimedia.org/P89103 and previous config saved to /var/cache/conftool/dbconfig/20260227-091847-marostegui.json |
[production] |
| 09:18 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2193.codfw.wmnet with reason: Maintenance |
[production] |
| 09:18 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2180 (T418465)', diff saved to https://phabricator.wikimedia.org/P89102 and previous config saved to /var/cache/conftool/dbconfig/20260227-091822-marostegui.json |
[production] |
| 09:06 |
<aokoth@cumin1003> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 09:03 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P89101 and previous config saved to /var/cache/conftool/dbconfig/20260227-090314-marostegui.json |
[production] |
| 08:59 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1027.eqiad.wmnet with reason: host reimage |
[production] |
| 08:55 |
<marostegui@cumin1003> |
START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy1027.eqiad.wmnet with reason: host reimage |
[production] |
| 08:51 |
<aokoth@cumin1003> |
END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 08:48 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P89100 and previous config saved to /var/cache/conftool/dbconfig/20260227-084806-marostegui.json |
[production] |
| 08:45 |
<Emperor> |
restart corto on alert1002 |
[production] |
| 08:40 |
<aokoth@cumin1003> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 08:39 |
<marostegui@cumin1003> |
START - Cookbook sre.hosts.reimage for host dbproxy1027.eqiad.wmnet with OS trixie |
[production] |
| 08:32 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2180 (T418465)', diff saved to https://phabricator.wikimedia.org/P89099 and previous config saved to /var/cache/conftool/dbconfig/20260227-083257-marostegui.json |
[production] |
| 08:32 |
<aokoth@cumin1003> |
END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 08:32 |
<aokoth@cumin1003> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 08:30 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2180 (T418465)', diff saved to https://phabricator.wikimedia.org/P89098 and previous config saved to /var/cache/conftool/dbconfig/20260227-083043-marostegui.json |
[production] |
| 08:30 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2180.codfw.wmnet with reason: Maintenance |
[production] |
| 08:30 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2169 (T418465)', diff saved to https://phabricator.wikimedia.org/P89097 and previous config saved to /var/cache/conftool/dbconfig/20260227-083018-marostegui.json |
[production] |
| 08:22 |
<aokoth@cumin1003> |
END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 08:22 |
<aokoth@cumin1003> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 08:19 |
<aokoth@cumin1003> |
END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 08:19 |
<aokoth@cumin1003> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 08:15 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P89096 and previous config saved to /var/cache/conftool/dbconfig/20260227-081510-marostegui.json |
[production] |
| 08:14 |
<aokoth@cumin1003> |
END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 08:14 |
<aokoth@cumin1003> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 08:05 |
<brouberol@deploy2002> |
helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 08:04 |
<brouberol@deploy2002> |
helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 08:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P89095 and previous config saved to /var/cache/conftool/dbconfig/20260227-080002-marostegui.json |
[production] |
| 07:44 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2169 (T418465)', diff saved to https://phabricator.wikimedia.org/P89094 and previous config saved to /var/cache/conftool/dbconfig/20260227-074454-marostegui.json |
[production] |
| 07:39 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2169 (T418465)', diff saved to https://phabricator.wikimedia.org/P89092 and previous config saved to /var/cache/conftool/dbconfig/20260227-073957-marostegui.json |
[production] |
| 07:39 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2169.codfw.wmnet with reason: Maintenance |
[production] |
| 07:39 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2158 (T418465)', diff saved to https://phabricator.wikimedia.org/P89091 and previous config saved to /var/cache/conftool/dbconfig/20260227-073943-marostegui.json |
[production] |
| 07:38 |
<aokoth@cumin1003> |
END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 07:27 |
<aokoth@cumin1003> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 07:25 |
<aokoth@cumin1003> |
END (ERROR) - Cookbook sre.gitlab.upgrade (exit_code=97) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 07:24 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P89090 and previous config saved to /var/cache/conftool/dbconfig/20260227-072434-marostegui.json |
[production] |
| 07:19 |
<aokoth@cumin1003> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 07:18 |
<aokoth@cumin1003> |
END (ERROR) - Cookbook sre.gitlab.upgrade (exit_code=97) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 07:15 |
<aokoth@cumin1003> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 07:09 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P89089 and previous config saved to /var/cache/conftool/dbconfig/20260227-070926-marostegui.json |
[production] |
| 06:54 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2158 (T418465)', diff saved to https://phabricator.wikimedia.org/P89088 and previous config saved to /var/cache/conftool/dbconfig/20260227-065417-marostegui.json |
[production] |
| 06:49 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2158 (T418465)', diff saved to https://phabricator.wikimedia.org/P89087 and previous config saved to /var/cache/conftool/dbconfig/20260227-064922-marostegui.json |
[production] |
| 06:49 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2158.codfw.wmnet with reason: Maintenance |
[production] |
| 06:48 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2151 (T418465)', diff saved to https://phabricator.wikimedia.org/P89086 and previous config saved to /var/cache/conftool/dbconfig/20260227-064856-marostegui.json |
[production] |
| 06:35 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1162: Repooling after switchover |
[production] |
| 06:33 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P89084 and previous config saved to /var/cache/conftool/dbconfig/20260227-063348-marostegui.json |
[production] |
| 06:18 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P89082 and previous config saved to /var/cache/conftool/dbconfig/20260227-061840-marostegui.json |
[production] |
| 06:07 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1244.eqiad.wmnet with reason: Maintenance |
[production] |