|
2026-01-29
ยง
|
| 09:36 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P88097 and previous config saved to /var/cache/conftool/dbconfig/20260129-093644-marostegui.json |
[production] |
| 09:28 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1185 (T415786)', diff saved to https://phabricator.wikimedia.org/P88096 and previous config saved to /var/cache/conftool/dbconfig/20260129-092806-marostegui.json |
[production] |
| 09:27 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1185.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:27 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T415786)', diff saved to https://phabricator.wikimedia.org/P88095 and previous config saved to /var/cache/conftool/dbconfig/20260129-092741-marostegui.json |
[production] |
| 09:21 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P88094 and previous config saved to /var/cache/conftool/dbconfig/20260129-092135-marostegui.json |
[production] |
| 09:12 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P88093 and previous config saved to /var/cache/conftool/dbconfig/20260129-091232-marostegui.json |
[production] |
| 09:06 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2171 (T415786)', diff saved to https://phabricator.wikimedia.org/P88092 and previous config saved to /var/cache/conftool/dbconfig/20260129-090628-marostegui.json |
[production] |
| 08:57 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P88091 and previous config saved to /var/cache/conftool/dbconfig/20260129-085724-marostegui.json |
[production] |
| 08:44 |
<kharlan@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1234918|BlockUtils: Remove x-provenance (T415354)]] (duration: 07m 45s) |
[production] |
| 08:42 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1161 (T415786)', diff saved to https://phabricator.wikimedia.org/P88089 and previous config saved to /var/cache/conftool/dbconfig/20260129-084216-marostegui.json |
[production] |
| 08:40 |
<kharlan@deploy2002> |
kharlan: Continuing with sync |
[production] |
| 08:38 |
<kharlan@deploy2002> |
kharlan: Backport for [[gerrit:1234918|BlockUtils: Remove x-provenance (T415354)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 08:36 |
<kharlan@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1234918|BlockUtils: Remove x-provenance (T415354)]] |
[production] |
| 08:27 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2171 (T415786)', diff saved to https://phabricator.wikimedia.org/P88088 and previous config saved to /var/cache/conftool/dbconfig/20260129-082722-marostegui.json |
[production] |
| 08:27 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2171.codfw.wmnet with reason: Maintenance |
[production] |
| 08:26 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2157 (T415786)', diff saved to https://phabricator.wikimedia.org/P88087 and previous config saved to /var/cache/conftool/dbconfig/20260129-082656-marostegui.json |
[production] |
| 08:11 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P88086 and previous config saved to /var/cache/conftool/dbconfig/20260129-081148-marostegui.json |
[production] |
| 08:03 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1161 (T415786)', diff saved to https://phabricator.wikimedia.org/P88085 and previous config saved to /var/cache/conftool/dbconfig/20260129-080327-marostegui.json |
[production] |
| 08:03 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:03 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1161.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:02 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1159 (T415786)', diff saved to https://phabricator.wikimedia.org/P88084 and previous config saved to /var/cache/conftool/dbconfig/20260129-080251-marostegui.json |
[production] |
| 07:56 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P88083 and previous config saved to /var/cache/conftool/dbconfig/20260129-075639-marostegui.json |
[production] |
| 07:47 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P88082 and previous config saved to /var/cache/conftool/dbconfig/20260129-074742-marostegui.json |
[production] |
| 07:41 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2157 (T415786)', diff saved to https://phabricator.wikimedia.org/P88081 and previous config saved to /var/cache/conftool/dbconfig/20260129-074130-marostegui.json |
[production] |
| 07:32 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P88080 and previous config saved to /var/cache/conftool/dbconfig/20260129-073232-marostegui.json |
[production] |
| 07:21 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.newpool (exit_code=0) pool db2212: After schema change |
[production] |
| 07:17 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1159 (T415786)', diff saved to https://phabricator.wikimedia.org/P88078 and previous config saved to /var/cache/conftool/dbconfig/20260129-071724-marostegui.json |
[production] |
| 06:59 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1201.eqiad.wmnet with reason: Schema change on db1201 |
[production] |
| 06:58 |
<marostegui> |
Starting s6 eqiad failover from db1201 to db1173 - T415861 |
[production] |
| 06:58 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db1201 T415861', diff saved to https://phabricator.wikimedia.org/P88076 and previous config saved to /var/cache/conftool/dbconfig/20260129-065838-marostegui.json |
[production] |
| 06:57 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Promote db1173 to s6 primary T415861', diff saved to https://phabricator.wikimedia.org/P88075 and previous config saved to /var/cache/conftool/dbconfig/20260129-065753-marostegui.json |
[production] |
| 06:55 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 22 hosts with reason: Primary switchover s6 T415861 |
[production] |
| 06:55 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Set db1173 with weight 0 T415861', diff saved to https://phabricator.wikimedia.org/P88074 and previous config saved to /var/cache/conftool/dbconfig/20260129-065528-marostegui.json |
[production] |
| 06:38 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1159 (T415786)', diff saved to https://phabricator.wikimedia.org/P88072 and previous config saved to /var/cache/conftool/dbconfig/20260129-063820-marostegui.json |
[production] |
| 06:38 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2157 (T415786)', diff saved to https://phabricator.wikimedia.org/P88071 and previous config saved to /var/cache/conftool/dbconfig/20260129-063813-marostegui.json |
[production] |
| 06:38 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1159.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:38 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2157.codfw.wmnet with reason: Maintenance |
[production] |
| 06:35 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.newpool pool db2212: After schema change |
[production] |
| 04:34 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2229 (T415786)', diff saved to https://phabricator.wikimedia.org/P88069 and previous config saved to /var/cache/conftool/dbconfig/20260129-043443-marostegui.json |
[production] |
| 04:19 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P88068 and previous config saved to /var/cache/conftool/dbconfig/20260129-041934-marostegui.json |
[production] |
| 04:04 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P88067 and previous config saved to /var/cache/conftool/dbconfig/20260129-040426-marostegui.json |
[production] |
| 03:49 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2229 (T415786)', diff saved to https://phabricator.wikimedia.org/P88066 and previous config saved to /var/cache/conftool/dbconfig/20260129-034917-marostegui.json |
[production] |
| 03:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2229 (T415786)', diff saved to https://phabricator.wikimedia.org/P88065 and previous config saved to /var/cache/conftool/dbconfig/20260129-030008-marostegui.json |
[production] |
| 03:00 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2229.codfw.wmnet with reason: Maintenance |
[production] |
| 02:59 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2224 (T415786)', diff saved to https://phabricator.wikimedia.org/P88064 and previous config saved to /var/cache/conftool/dbconfig/20260129-025943-marostegui.json |
[production] |
| 02:44 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P88063 and previous config saved to /var/cache/conftool/dbconfig/20260129-024435-marostegui.json |
[production] |
| 02:29 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P88062 and previous config saved to /var/cache/conftool/dbconfig/20260129-022926-marostegui.json |
[production] |
| 02:14 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2224 (T415786)', diff saved to https://phabricator.wikimedia.org/P88061 and previous config saved to /var/cache/conftool/dbconfig/20260129-021418-marostegui.json |
[production] |
| 02:13 |
<mwpresync@deploy2002> |
Finished scap build-images: Publishing wmf/next image (duration: 12m 44s) |
[production] |
| 02:01 |
<mwpresync@deploy2002> |
Started scap build-images: Publishing wmf/next image |
[production] |