|
2026-02-27
ยง
|
| 08:19 |
<aokoth@cumin1003> |
END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 08:19 |
<aokoth@cumin1003> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 08:15 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P89096 and previous config saved to /var/cache/conftool/dbconfig/20260227-081510-marostegui.json |
[production] |
| 08:14 |
<aokoth@cumin1003> |
END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 08:14 |
<aokoth@cumin1003> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 08:05 |
<brouberol@deploy2002> |
helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 08:04 |
<brouberol@deploy2002> |
helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 08:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P89095 and previous config saved to /var/cache/conftool/dbconfig/20260227-080002-marostegui.json |
[production] |
| 07:44 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2169 (T418465)', diff saved to https://phabricator.wikimedia.org/P89094 and previous config saved to /var/cache/conftool/dbconfig/20260227-074454-marostegui.json |
[production] |
| 07:39 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2169 (T418465)', diff saved to https://phabricator.wikimedia.org/P89092 and previous config saved to /var/cache/conftool/dbconfig/20260227-073957-marostegui.json |
[production] |
| 07:39 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2169.codfw.wmnet with reason: Maintenance |
[production] |
| 07:39 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2158 (T418465)', diff saved to https://phabricator.wikimedia.org/P89091 and previous config saved to /var/cache/conftool/dbconfig/20260227-073943-marostegui.json |
[production] |
| 07:38 |
<aokoth@cumin1003> |
END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 07:27 |
<aokoth@cumin1003> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 07:25 |
<aokoth@cumin1003> |
END (ERROR) - Cookbook sre.gitlab.upgrade (exit_code=97) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 07:24 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P89090 and previous config saved to /var/cache/conftool/dbconfig/20260227-072434-marostegui.json |
[production] |
| 07:19 |
<aokoth@cumin1003> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 07:18 |
<aokoth@cumin1003> |
END (ERROR) - Cookbook sre.gitlab.upgrade (exit_code=97) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 07:15 |
<aokoth@cumin1003> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T418483 |
[production] |
| 07:09 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P89089 and previous config saved to /var/cache/conftool/dbconfig/20260227-070926-marostegui.json |
[production] |
| 06:54 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2158 (T418465)', diff saved to https://phabricator.wikimedia.org/P89088 and previous config saved to /var/cache/conftool/dbconfig/20260227-065417-marostegui.json |
[production] |
| 06:49 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2158 (T418465)', diff saved to https://phabricator.wikimedia.org/P89087 and previous config saved to /var/cache/conftool/dbconfig/20260227-064922-marostegui.json |
[production] |
| 06:49 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2158.codfw.wmnet with reason: Maintenance |
[production] |
| 06:48 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2151 (T418465)', diff saved to https://phabricator.wikimedia.org/P89086 and previous config saved to /var/cache/conftool/dbconfig/20260227-064856-marostegui.json |
[production] |
| 06:35 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1162: Repooling after switchover |
[production] |
| 06:33 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P89084 and previous config saved to /var/cache/conftool/dbconfig/20260227-063348-marostegui.json |
[production] |
| 06:18 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P89082 and previous config saved to /var/cache/conftool/dbconfig/20260227-061840-marostegui.json |
[production] |
| 06:07 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1244.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:05 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db1244 T418079', diff saved to https://phabricator.wikimedia.org/P89081 and previous config saved to /var/cache/conftool/dbconfig/20260227-060534-marostegui.json |
[production] |
| 06:04 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Promote db1160 to s4 primary T418079', diff saved to https://phabricator.wikimedia.org/P89080 and previous config saved to /var/cache/conftool/dbconfig/20260227-060455-marostegui.json |
[production] |
| 06:03 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2151 (T418465)', diff saved to https://phabricator.wikimedia.org/P89078 and previous config saved to /var/cache/conftool/dbconfig/20260227-060331-marostegui.json |
[production] |
| 06:00 |
<marostegui> |
revert: Failover m5-master T401966 |
[production] |
| 06:00 |
<marostegui> |
Starting s4 eqiad failover from db1244 to db1160 - T418079 |
[production] |
| 05:59 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 42 hosts with reason: Primary switchover s4 T418079 |
[production] |
| 05:58 |
<marostegui@dns1006> |
END - running authdns-update |
[production] |
| 05:58 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Set db1160 with weight 0 T418079', diff saved to https://phabricator.wikimedia.org/P89077 and previous config saved to /var/cache/conftool/dbconfig/20260227-055845-marostegui.json |
[production] |
| 05:58 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2151 (T418465)', diff saved to https://phabricator.wikimedia.org/P89076 and previous config saved to /var/cache/conftool/dbconfig/20260227-055835-marostegui.json |
[production] |
| 05:58 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2151.codfw.wmnet with reason: Maintenance |
[production] |
| 05:57 |
<marostegui@dns1006> |
START - running authdns-update |
[production] |
| 05:56 |
<marostegui@dns1006> |
END - running authdns-update |
[production] |
| 05:55 |
<marostegui> |
Failover m5-master T401966 |
[production] |
| 05:55 |
<marostegui@dns1006> |
START - running authdns-update |
[production] |
| 05:51 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1173.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:49 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool pool db1162: Repooling after switchover |
[production] |
| 05:49 |
<marostegui@cumin1003> |
END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1162: Repooling after switchover |
[production] |
| 05:48 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool pool db1162: Repooling after switchover |
[production] |
| 05:48 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db1162 T418553', diff saved to https://phabricator.wikimedia.org/P89074 and previous config saved to /var/cache/conftool/dbconfig/20260227-054833-marostegui.json |
[production] |
| 05:47 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Promote db1222 to s2 primary T418553', diff saved to https://phabricator.wikimedia.org/P89073 and previous config saved to /var/cache/conftool/dbconfig/20260227-054750-marostegui.json |
[production] |
| 05:47 |
<marostegui> |
Starting s2 eqiad failover from db1162 to db1222 - T418553 |
[production] |
| 05:44 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s2 T418553 |
[production] |