|
2026-01-20
§
|
| 07:45 |
<cgoubert@deploy2002> |
helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 07:42 |
<cgoubert@deploy2002> |
helmfile [staging-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 07:42 |
<cgoubert@deploy2002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 07:41 |
<cgoubert@deploy2002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 06:49 |
<marostegui> |
Deploy schema change on s8 sanitarium master - s8 wikireplicas will be lagging for many hours T411164 T411163 |
[production] |
| 06:48 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2152 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87781 and previous config saved to /var/cache/conftool/dbconfig/20260120-064801-marostegui.json |
[production] |
| 06:47 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2152.codfw.wmnet with reason: Maintenance |
[production] |
| 06:47 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1167 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87780 and previous config saved to /var/cache/conftool/dbconfig/20260120-064708-marostegui.json |
[production] |
| 06:47 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:46 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:37 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1160.eqiad.wmnet with reason: Schema change |
[production] |
| 06:29 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance |
[production] |
| 06:27 |
<marostegui@dns1006> |
END - running authdns-update |
[production] |
| 06:26 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db2179 T414543', diff saved to https://phabricator.wikimedia.org/P87779 and previous config saved to /var/cache/conftool/dbconfig/20260120-062653-marostegui.json |
[production] |
| 06:26 |
<marostegui@dns1006> |
START - running authdns-update |
[production] |
| 06:25 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Promote db2240 to s4 primary and set section read-write T414543', diff saved to https://phabricator.wikimedia.org/P87778 and previous config saved to /var/cache/conftool/dbconfig/20260120-062551-marostegui.json |
[production] |
| 06:25 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Set s4 codfw as read-only for maintenance - T414543', diff saved to https://phabricator.wikimedia.org/P87777 and previous config saved to /var/cache/conftool/dbconfig/20260120-062527-marostegui.json |
[production] |
| 06:25 |
<marostegui> |
Starting s4 codfw failover from db2179 to db2240 - T414543 |
[production] |
| 06:19 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 42 hosts with reason: Primary switchover s4 T414543 |
[production] |
| 06:18 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Set db2240 with weight 0 T414543', diff saved to https://phabricator.wikimedia.org/P87776 and previous config saved to /var/cache/conftool/dbconfig/20260120-061840-marostegui.json |
[production] |
| 05:03 |
<mwpresync@deploy2002> |
Pruned MediaWiki: 1.46.0-wmf.7 (duration: 03m 23s) |
[production] |
| 04:47 |
<mwpresync@deploy2002> |
Finished scap sync-world: testwikis to 1.46.0-wmf.12 refs T413803 (duration: 44m 14s) |
[production] |
| 04:31 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1219 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87775 and previous config saved to /var/cache/conftool/dbconfig/20260120-043145-marostegui.json |
[production] |
| 04:31 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance |
[production] |
| 04:31 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1218 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87774 and previous config saved to /var/cache/conftool/dbconfig/20260120-043120-marostegui.json |
[production] |
| 04:21 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P87773 and previous config saved to /var/cache/conftool/dbconfig/20260120-042112-marostegui.json |
[production] |
| 04:11 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P87772 and previous config saved to /var/cache/conftool/dbconfig/20260120-041104-marostegui.json |
[production] |
| 04:03 |
<mwpresync@deploy2002> |
Started scap sync-world: testwikis to 1.46.0-wmf.12 refs T413803 |
[production] |
| 04:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1218 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87771 and previous config saved to /var/cache/conftool/dbconfig/20260120-040055-marostegui.json |
[production] |
| 03:01 |
<ladsgroup@cumin1003> |
dbctl commit (dc=all): 'Depooling db2175 (T410589)', diff saved to https://phabricator.wikimedia.org/P87770 and previous config saved to /var/cache/conftool/dbconfig/20260120-030136-ladsgroup.json |
[production] |
| 03:01 |
<ladsgroup@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance |
[production] |
| 03:01 |
<ladsgroup@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2148 (T410589)', diff saved to https://phabricator.wikimedia.org/P87769 and previous config saved to /var/cache/conftool/dbconfig/20260120-030112-ladsgroup.json |
[production] |
| 02:51 |
<ladsgroup@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P87768 and previous config saved to /var/cache/conftool/dbconfig/20260120-025103-ladsgroup.json |
[production] |
| 02:40 |
<ladsgroup@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P87767 and previous config saved to /var/cache/conftool/dbconfig/20260120-024056-ladsgroup.json |
[production] |
| 02:30 |
<ladsgroup@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2148 (T410589)', diff saved to https://phabricator.wikimedia.org/P87766 and previous config saved to /var/cache/conftool/dbconfig/20260120-023047-ladsgroup.json |
[production] |
| 01:01 |
<mwpresync@deploy2002> |
Finished scap build-images: Publishing wmf/next image (duration: 01m 03s) |
[production] |
| 01:00 |
<mwpresync@deploy2002> |
Started scap build-images: Publishing wmf/next image |
[production] |
|
2026-01-19
§
|
| 21:24 |
<tgr@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1227493|enwikiquote: Add autopatroller protection option (T414711)]], [[gerrit:1226903|debug: Add X-Provenance header to Logstash (T412396)]], [[gerrit:1226366|Urwikiquote: restore flipped icon (T413592)]] (duration: 09m 58s) |
[production] |
| 21:20 |
<tgr@deploy2002> |
seawolf35gerrit, pppery, tgr: Continuing with sync |
[production] |
| 21:16 |
<tgr@deploy2002> |
seawolf35gerrit, pppery, tgr: Backport for [[gerrit:1227493|enwikiquote: Add autopatroller protection option (T414711)]], [[gerrit:1226903|debug: Add X-Provenance header to Logstash (T412396)]], [[gerrit:1226366|Urwikiquote: restore flipped icon (T413592)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 21:14 |
<tgr@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1227493|enwikiquote: Add autopatroller protection option (T414711)]], [[gerrit:1226903|debug: Add X-Provenance header to Logstash (T412396)]], [[gerrit:1226366|Urwikiquote: restore flipped icon (T413592)]] |
[production] |
| 21:01 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2188 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87763 and previous config saved to /var/cache/conftool/dbconfig/20260119-210153-marostegui.json |
[production] |
| 21:01 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2188.codfw.wmnet with reason: Maintenance |
[production] |
| 21:01 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2176 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87762 and previous config saved to /var/cache/conftool/dbconfig/20260119-210128-marostegui.json |
[production] |
| 20:51 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P87761 and previous config saved to /var/cache/conftool/dbconfig/20260119-205120-marostegui.json |
[production] |
| 20:41 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P87760 and previous config saved to /var/cache/conftool/dbconfig/20260119-204112-marostegui.json |
[production] |
| 20:31 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2176 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87759 and previous config saved to /var/cache/conftool/dbconfig/20260119-203104-marostegui.json |
[production] |
| 20:21 |
<brett@dns1006> |
END - running authdns-update |
[production] |
| 20:20 |
<brett@dns1006> |
START - running authdns-update |
[production] |
| 19:02 |
<wmbot~lucaswerkmeister@tools-bastion-15> |
deployed 4c97ea39c4 (l10n updates: gl, he) |
[tools.ranker] |