|
2025-07-16
ยง
|
| 07:55 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2241 T399456', diff saved to https://phabricator.wikimedia.org/P79182 and previous config saved to /var/cache/conftool/dbconfig/20250716-075534-marostegui.json |
[production] |
| 07:55 |
<elukey@cumin1003> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2010.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 07:54 |
<marostegui> |
Starting x3 codfw failover from db2241 to db2162 - T399456 |
[production] |
| 07:54 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote db2162 to x3 primary T399456', diff saved to https://phabricator.wikimedia.org/P79181 and previous config saved to /var/cache/conftool/dbconfig/20250716-075448-marostegui.json |
[production] |
| 07:54 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.provision for host sretest2010.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 07:50 |
<jelto@cumin1003> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab1004.wikimedia.org with reason: host reimage |
[production] |
| 07:49 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Primary switchover x3 T399456 |
[production] |
| 07:49 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Set db2162 with weight 0 T399456', diff saved to https://phabricator.wikimedia.org/P79180 and previous config saved to /var/cache/conftool/dbconfig/20250716-074931-root.json |
[production] |
| 07:48 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1257 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P79179 and previous config saved to /var/cache/conftool/dbconfig/20250716-074855-root.json |
[production] |
| 07:46 |
<jelto@cumin1003> |
START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab1004.wikimedia.org with reason: host reimage |
[production] |
| 07:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P79178 and previous config saved to /var/cache/conftool/dbconfig/20250716-074538-marostegui.json |
[production] |
| 07:33 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1257 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P79177 and previous config saved to /var/cache/conftool/dbconfig/20250716-073349-root.json |
[production] |
| 07:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T399249)', diff saved to https://phabricator.wikimedia.org/P79176 and previous config saved to /var/cache/conftool/dbconfig/20250716-073031-marostegui.json |
[production] |
| 07:29 |
<jelto@cumin1003> |
START - Cookbook sre.hosts.reimage for host gitlab1004.wikimedia.org with OS bookworm |
[production] |
| 07:22 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2149 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P79175 and previous config saved to /var/cache/conftool/dbconfig/20250716-072205-root.json |
[production] |
| 07:18 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1257 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P79174 and previous config saved to /var/cache/conftool/dbconfig/20250716-071844-root.json |
[production] |
| 07:07 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2149 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P79173 and previous config saved to /var/cache/conftool/dbconfig/20250716-070659-root.json |
[production] |
| 07:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1257 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P79172 and previous config saved to /var/cache/conftool/dbconfig/20250716-070338-root.json |
[production] |
| 07:01 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1161 (T399249)', diff saved to https://phabricator.wikimedia.org/P79171 and previous config saved to /var/cache/conftool/dbconfig/20250716-070130-marostegui.json |
[production] |
| 07:01 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:01 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1161.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:01 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1159 (T399249)', diff saved to https://phabricator.wikimedia.org/P79170 and previous config saved to /var/cache/conftool/dbconfig/20250716-070101-marostegui.json |
[production] |
| 06:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1257 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P79169 and previous config saved to /var/cache/conftool/dbconfig/20250716-065626-marostegui.json |
[production] |
| 06:56 |
<root@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1257.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:51 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2149 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P79168 and previous config saved to /var/cache/conftool/dbconfig/20250716-065152-root.json |
[production] |
| 06:47 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1256 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P79167 and previous config saved to /var/cache/conftool/dbconfig/20250716-064705-root.json |
[production] |
| 06:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P79166 and previous config saved to /var/cache/conftool/dbconfig/20250716-064553-marostegui.json |
[production] |
| 06:44 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Upgrade x3 codfw master |
[production] |
| 06:39 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 23 hosts with reason: Upgrade s6 codfw master |
[production] |
| 06:36 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2149 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P79165 and previous config saved to /var/cache/conftool/dbconfig/20250716-063646-root.json |
[production] |
| 06:32 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1256 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P79164 and previous config saved to /var/cache/conftool/dbconfig/20250716-063159-root.json |
[production] |
| 06:30 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P79163 and previous config saved to /var/cache/conftool/dbconfig/20250716-063046-marostegui.json |
[production] |
| 06:25 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2149 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P79162 and previous config saved to /var/cache/conftool/dbconfig/20250716-062537-marostegui.json |
[production] |
| 06:25 |
<root@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2149.codfw.wmnet with reason: Maintenance |
[production] |
| 06:19 |
<marostegui> |
Poweroff pc2015 for 10G migration T378715 |
[production] |
| 06:16 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1256 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P79161 and previous config saved to /var/cache/conftool/dbconfig/20250716-061653-root.json |
[production] |
| 06:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1159 (T399249)', diff saved to https://phabricator.wikimedia.org/P79160 and previous config saved to /var/cache/conftool/dbconfig/20250716-061539-marostegui.json |
[production] |
| 06:09 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 6:00:00 on pc2015.codfw.wmnet,pc1015.eqiad.wmnet with reason: maintenance |
[production] |
| 06:03 |
<marostegui> |
Restart mariadb on pc5 T399540 |
[production] |
| 06:01 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1256 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P79159 and previous config saved to /var/cache/conftool/dbconfig/20250716-060148-root.json |
[production] |
| 06:00 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) |
[production] |
| 05:59 |
<marostegui@cumin1002> |
START - Cookbook sre.mysql.parsercache |
[production] |
| 05:54 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db1256 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P79157 and previous config saved to /var/cache/conftool/dbconfig/20250716-055408-marostegui.json |
[production] |
| 05:54 |
<root@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1256.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:46 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1159 (T399249)', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20250716-054645-marostegui.json |
[production] |
| 05:46 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1159.eqiad.wmnet with reason: Maintenance |
[production] |
| 03:31 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2228 (T399249)', diff saved to https://phabricator.wikimedia.org/P79156 and previous config saved to /var/cache/conftool/dbconfig/20250716-033133-marostegui.json |
[production] |
| 03:16 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P79155 and previous config saved to /var/cache/conftool/dbconfig/20250716-031626-marostegui.json |
[production] |
| 03:01 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P79154 and previous config saved to /var/cache/conftool/dbconfig/20250716-030119-marostegui.json |
[production] |
| 02:46 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2228 (T399249)', diff saved to https://phabricator.wikimedia.org/P79153 and previous config saved to /var/cache/conftool/dbconfig/20250716-024611-marostegui.json |
[production] |