|
2026-06-04
ยง
|
| 06:16 |
<cwilliams@cumin1003> |
START - Cookbook sre.hosts.reimage for host db1255.eqiad.wmnet with OS trixie |
[production] |
| 06:15 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2191.codfw.wmnet with reason: host reimage |
[production] |
| 06:13 |
<cwilliams@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1255: Upgrading db1255.eqiad.wmnet |
[production] |
| 06:12 |
<cwilliams@cumin1003> |
START - Cookbook sre.mysql.depool depool db1255: Upgrading db1255.eqiad.wmnet |
[production] |
| 06:12 |
<cwilliams@cumin1003> |
START - Cookbook sre.mysql.major-upgrade |
[production] |
| 06:11 |
<marostegui@cumin1003> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db2191.codfw.wmnet with reason: host reimage |
[production] |
| 06:04 |
<cwilliams@cumin1003> |
dbctl commit (dc=all): 'Depool db1255 T427895', diff saved to https://phabricator.wikimedia.org/P93836 and previous config saved to /var/cache/conftool/dbconfig/20260604-060428-cwilliams.json |
[production] |
| 06:03 |
<cwilliams@dns1004> |
END - running authdns-update |
[production] |
| 06:02 |
<cwilliams@dns1004> |
START - running authdns-update |
[production] |
| 05:54 |
<cwilliams@cumin1003> |
dbctl commit (dc=all): 'Promote db1258 to x3 primary and set section read-write T427895', diff saved to https://phabricator.wikimedia.org/P93835 and previous config saved to /var/cache/conftool/dbconfig/20260604-055429-cwilliams.json |
[production] |
| 05:53 |
<cwilliams@cumin1003> |
dbctl commit (dc=all): 'Set x3 eqiad as read-only for maintenance - T427895', diff saved to https://phabricator.wikimedia.org/P93834 and previous config saved to /var/cache/conftool/dbconfig/20260604-055346-cwilliams.json |
[production] |
| 05:53 |
<cezmunsta> |
Starting x3 eqiad failover from db1255 to db1258 - T427895 |
[production] |
| 05:52 |
<marostegui@cumin1003> |
START - Cookbook sre.hosts.reimage for host db2191.codfw.wmnet with OS trixie |
[production] |
| 05:50 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2191: Upgrading db2191.codfw.wmnet |
[production] |
| 05:50 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.depool depool db2191: Upgrading db2191.codfw.wmnet |
[production] |
| 05:50 |
<cwilliams@cumin1003> |
dbctl commit (dc=all): 'Set db1258 with weight 0 T427895', diff saved to https://phabricator.wikimedia.org/P93833 and previous config saved to /var/cache/conftool/dbconfig/20260604-055021-cwilliams.json |
[production] |
| 05:50 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.major-upgrade |
[production] |
| 05:50 |
<cwilliams@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Primary switchover x3 T427895 |
[production] |
| 05:48 |
<kevinbazira@deploy1003> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . |
[production] |
| 05:46 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db2191 T428120', diff saved to https://phabricator.wikimedia.org/P93832 and previous config saved to /var/cache/conftool/dbconfig/20260604-054614-marostegui.json |
[production] |
| 05:45 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Promote db2215 to x1 primary T428120', diff saved to https://phabricator.wikimedia.org/P93831 and previous config saved to /var/cache/conftool/dbconfig/20260604-054528-marostegui.json |
[production] |
| 05:44 |
<marostegui> |
Starting x1 codfw failover from db2191 to db2215 - T428120 |
[production] |
| 05:27 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x1 T428120 |
[production] |
| 05:27 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Set db2215 with weight 0 T428120', diff saved to https://phabricator.wikimedia.org/P93830 and previous config saved to /var/cache/conftool/dbconfig/20260604-052722-marostegui.json |
[production] |
| 05:19 |
<kevinbazira@deploy1003> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . |
[production] |
| 03:45 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1263 (T426633)', diff saved to https://phabricator.wikimedia.org/P93829 and previous config saved to /var/cache/conftool/dbconfig/20260604-034546-fceratto.json |
[production] |
| 03:35 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P93828 and previous config saved to /var/cache/conftool/dbconfig/20260604-033538-fceratto.json |
[production] |
| 03:25 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P93827 and previous config saved to /var/cache/conftool/dbconfig/20260604-032531-fceratto.json |
[production] |
| 03:15 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1263 (T426633)', diff saved to https://phabricator.wikimedia.org/P93826 and previous config saved to /var/cache/conftool/dbconfig/20260604-031523-fceratto.json |
[production] |
| 03:07 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Depooling db1263 (T426633)', diff saved to https://phabricator.wikimedia.org/P93825 and previous config saved to /var/cache/conftool/dbconfig/20260604-030710-fceratto.json |
[production] |
| 03:07 |
<fceratto@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1263.eqiad.wmnet with reason: Maintenance |
[production] |
| 03:06 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1262 (T426633)', diff saved to https://phabricator.wikimedia.org/P93824 and previous config saved to /var/cache/conftool/dbconfig/20260604-030642-fceratto.json |
[production] |
| 02:56 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P93823 and previous config saved to /var/cache/conftool/dbconfig/20260604-025634-fceratto.json |
[production] |
| 02:46 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P93822 and previous config saved to /var/cache/conftool/dbconfig/20260604-024627-fceratto.json |
[production] |
| 02:36 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1262 (T426633)', diff saved to https://phabricator.wikimedia.org/P93821 and previous config saved to /var/cache/conftool/dbconfig/20260604-023619-fceratto.json |
[production] |
| 02:28 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Depooling db1262 (T426633)', diff saved to https://phabricator.wikimedia.org/P93820 and previous config saved to /var/cache/conftool/dbconfig/20260604-022809-fceratto.json |
[production] |
| 02:28 |
<fceratto@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Maintenance |
[production] |
| 02:27 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1261 (T426633)', diff saved to https://phabricator.wikimedia.org/P93819 and previous config saved to /var/cache/conftool/dbconfig/20260604-022742-fceratto.json |
[production] |
| 02:17 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P93818 and previous config saved to /var/cache/conftool/dbconfig/20260604-021734-fceratto.json |
[production] |
| 02:07 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P93817 and previous config saved to /var/cache/conftool/dbconfig/20260604-020726-fceratto.json |
[production] |
| 01:57 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1261 (T426633)', diff saved to https://phabricator.wikimedia.org/P93816 and previous config saved to /var/cache/conftool/dbconfig/20260604-015718-fceratto.json |
[production] |
| 01:49 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Depooling db1261 (T426633)', diff saved to https://phabricator.wikimedia.org/P93815 and previous config saved to /var/cache/conftool/dbconfig/20260604-014909-fceratto.json |
[production] |
| 01:49 |
<fceratto@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1261.eqiad.wmnet with reason: Maintenance |
[production] |
| 01:48 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1260 (T426633)', diff saved to https://phabricator.wikimedia.org/P93814 and previous config saved to /var/cache/conftool/dbconfig/20260604-014841-fceratto.json |
[production] |
| 01:38 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P93813 and previous config saved to /var/cache/conftool/dbconfig/20260604-013833-fceratto.json |
[production] |
| 01:28 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P93812 and previous config saved to /var/cache/conftool/dbconfig/20260604-012826-fceratto.json |
[production] |
| 01:18 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1260 (T426633)', diff saved to https://phabricator.wikimedia.org/P93811 and previous config saved to /var/cache/conftool/dbconfig/20260604-011818-fceratto.json |
[production] |
| 01:10 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Depooling db1260 (T426633)', diff saved to https://phabricator.wikimedia.org/P93810 and previous config saved to /var/cache/conftool/dbconfig/20260604-011005-fceratto.json |
[production] |
| 01:09 |
<fceratto@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1260.eqiad.wmnet with reason: Maintenance |
[production] |
| 01:09 |
<fceratto@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1252 (T426633)', diff saved to https://phabricator.wikimedia.org/P93809 and previous config saved to /var/cache/conftool/dbconfig/20260604-010937-fceratto.json |
[production] |