|
2026-01-12
ยง
|
| 10:41 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1203 gradually with 4 steps - repooling |
[production] |
| 10:34 |
<vgutierrez@cumin1003> |
END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp[7008,7016].*} and A:cp - test haproxy 2.8.18 upgrade (T414318) |
[production] |
| 10:22 |
<vgutierrez@cumin1003> |
START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp[7008,7016].*} and A:cp - test haproxy 2.8.18 upgrade (T414318) |
[production] |
| 10:20 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1169 (T413525)', diff saved to https://phabricator.wikimedia.org/P87313 and previous config saved to /var/cache/conftool/dbconfig/20260112-102044-marostegui.json |
[production] |
| 10:20 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1169.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:20 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2141.codfw.wmnet with reason: Maintenance |
[production] |
| 10:19 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool db2166 gradually with 4 steps - repooling |
[production] |
| 10:18 |
<vgutierrez> |
fetch HAProxy 2.8.18 on thirdparty/haproxy28-bullseye (apt.wm.o) - T414318 |
[production] |
| 10:15 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2166 (T413525)', diff saved to https://phabricator.wikimedia.org/P87310 and previous config saved to /var/cache/conftool/dbconfig/20260112-101520-marostegui.json |
[production] |
| 10:11 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool db1203 gradually with 4 steps - repooling |
[production] |
| 10:08 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2166 (T413525)', diff saved to https://phabricator.wikimedia.org/P87308 and previous config saved to /var/cache/conftool/dbconfig/20260112-100821-marostegui.json |
[production] |
| 10:08 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2166.codfw.wmnet with reason: Maintenance |
[production] |
| 10:07 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2165 (T413525)', diff saved to https://phabricator.wikimedia.org/P87307 and previous config saved to /var/cache/conftool/dbconfig/20260112-100756-marostegui.json |
[production] |
| 10:01 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1203 (T413525)', diff saved to https://phabricator.wikimedia.org/P87305 and previous config saved to /var/cache/conftool/dbconfig/20260112-100112-marostegui.json |
[production] |
| 09:57 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P87304 and previous config saved to /var/cache/conftool/dbconfig/20260112-095748-marostegui.json |
[production] |
| 09:56 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1203 (T413525)', diff saved to https://phabricator.wikimedia.org/P87303 and previous config saved to /var/cache/conftool/dbconfig/20260112-095610-marostegui.json |
[production] |
| 09:56 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1203.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:55 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1193 (T413525)', diff saved to https://phabricator.wikimedia.org/P87302 and previous config saved to /var/cache/conftool/dbconfig/20260112-095545-marostegui.json |
[production] |
| 09:47 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P87301 and previous config saved to /var/cache/conftool/dbconfig/20260112-094740-marostegui.json |
[production] |
| 09:47 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool db2249 slowly with 10 steps - repooling |
[production] |
| 09:45 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P87299 and previous config saved to /var/cache/conftool/dbconfig/20260112-094537-marostegui.json |
[production] |
| 09:37 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2165 (T413525)', diff saved to https://phabricator.wikimedia.org/P87298 and previous config saved to /var/cache/conftool/dbconfig/20260112-093731-marostegui.json |
[production] |
| 09:35 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P87297 and previous config saved to /var/cache/conftool/dbconfig/20260112-093528-marostegui.json |
[production] |
| 09:32 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2165 (T413525)', diff saved to https://phabricator.wikimedia.org/P87296 and previous config saved to /var/cache/conftool/dbconfig/20260112-093237-marostegui.json |
[production] |
| 09:32 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2165.codfw.wmnet with reason: Maintenance |
[production] |
| 09:32 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2164 (T413525)', diff saved to https://phabricator.wikimedia.org/P87295 and previous config saved to /var/cache/conftool/dbconfig/20260112-093223-marostegui.json |
[production] |
| 09:25 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1193 (T413525)', diff saved to https://phabricator.wikimedia.org/P87294 and previous config saved to /var/cache/conftool/dbconfig/20260112-092520-marostegui.json |
[production] |
| 09:22 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P87293 and previous config saved to /var/cache/conftool/dbconfig/20260112-092215-marostegui.json |
[production] |
| 09:20 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1193 (T413525)', diff saved to https://phabricator.wikimedia.org/P87292 and previous config saved to /var/cache/conftool/dbconfig/20260112-092016-marostegui.json |
[production] |
| 09:20 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1193.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:19 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1192 (T413525)', diff saved to https://phabricator.wikimedia.org/P87291 and previous config saved to /var/cache/conftool/dbconfig/20260112-091950-marostegui.json |
[production] |
| 09:12 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P87290 and previous config saved to /var/cache/conftool/dbconfig/20260112-091206-marostegui.json |
[production] |
| 09:10 |
<dzahn@dns1004> |
END - running authdns-update |
[production] |
| 09:10 |
<mutante> |
DNS - adding new language code 'kai' - https://en.wikipedia.org/wiki/Karai-karai |
[production] |
| 09:09 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P87289 and previous config saved to /var/cache/conftool/dbconfig/20260112-090942-marostegui.json |
[production] |
| 09:09 |
<dzahn@dns1004> |
START - running authdns-update |
[production] |
| 09:04 |
<bwojtowicz@deploy2002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . |
[production] |
| 09:01 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2164 (T413525)', diff saved to https://phabricator.wikimedia.org/P87288 and previous config saved to /var/cache/conftool/dbconfig/20260112-090158-marostegui.json |
[production] |
| 09:01 |
<bwojtowicz@deploy2002> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . |
[production] |
| 08:59 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P87287 and previous config saved to /var/cache/conftool/dbconfig/20260112-085934-marostegui.json |
[production] |
| 08:59 |
<XioNoX> |
eqiad pfw - remove old LVS BGP config (replaced by bird) - T414015 |
[production] |
| 08:55 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2164 (T413525)', diff saved to https://phabricator.wikimedia.org/P87286 and previous config saved to /var/cache/conftool/dbconfig/20260112-085459-marostegui.json |
[production] |
| 08:54 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2164.codfw.wmnet with reason: Maintenance |
[production] |
| 08:54 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2163 (T413525)', diff saved to https://phabricator.wikimedia.org/P87285 and previous config saved to /var/cache/conftool/dbconfig/20260112-085434-marostegui.json |
[production] |
| 08:49 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1192 (T413525)', diff saved to https://phabricator.wikimedia.org/P87284 and previous config saved to /var/cache/conftool/dbconfig/20260112-084926-marostegui.json |
[production] |
| 08:44 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P87283 and previous config saved to /var/cache/conftool/dbconfig/20260112-084426-marostegui.json |
[production] |
| 08:44 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1192 (T413525)', diff saved to https://phabricator.wikimedia.org/P87282 and previous config saved to /var/cache/conftool/dbconfig/20260112-084425-marostegui.json |
[production] |
| 08:44 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1192.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:44 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1178 (T413525)', diff saved to https://phabricator.wikimedia.org/P87281 and previous config saved to /var/cache/conftool/dbconfig/20260112-084401-marostegui.json |
[production] |
| 08:34 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P87280 and previous config saved to /var/cache/conftool/dbconfig/20260112-083418-marostegui.json |
[production] |