|
2025-08-28
ยง
|
| 13:10 |
<Emperor> |
restart swift on ms-fe2012 T360913 |
[production] |
| 13:08 |
<Emperor> |
restart swift on ms-fe2011 T360913 |
[production] |
| 13:06 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2225 (T401906)', diff saved to https://phabricator.wikimedia.org/P82015 and previous config saved to /var/cache/conftool/dbconfig/20250828-130654-fceratto.json |
[production] |
| 13:06 |
<lucaswerkmeister-wmde@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1182530|gotwiki: update wordmark and add tagline (T402706)]], [[gerrit:1182523|tlwiktionary: set sitename and projectnamespace (T402725)]] |
[production] |
| 13:04 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db2225 (T401906)', diff saved to https://phabricator.wikimedia.org/P82014 and previous config saved to /var/cache/conftool/dbconfig/20250828-130429-fceratto.json |
[production] |
| 13:04 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2225.codfw.wmnet with reason: Maintenance |
[production] |
| 13:01 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS bookworm |
[production] |
| 13:00 |
<dbrant@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply |
[production] |
| 13:00 |
<dbrant@deploy1003> |
helmfile [codfw] START helmfile.d/services/wikifeeds: apply |
[production] |
| 12:59 |
<dbrant@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply |
[production] |
| 12:59 |
<dbrant@deploy1003> |
helmfile [eqiad] START helmfile.d/services/wikifeeds: apply |
[production] |
| 12:58 |
<dbrant@deploy1003> |
helmfile [staging] DONE helmfile.d/services/wikifeeds: apply |
[production] |
| 12:58 |
<dbrant@deploy1003> |
helmfile [staging] START helmfile.d/services/wikifeeds: apply |
[production] |
| 12:57 |
<ladsgroup@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P82013 and previous config saved to /var/cache/conftool/dbconfig/20250828-125724-ladsgroup.json |
[production] |
| 12:51 |
<elukey@cumin1003> |
END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2009.codfw.wmnet with OS bookworm |
[production] |
| 12:49 |
<kharlan@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1182824|hCaptcha: Enable processing of the risk score (T379179)]] (duration: 14m 13s) |
[production] |
| 12:49 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS bookworm |
[production] |
| 12:44 |
<kharlan@deploy1003> |
kharlan: Continuing with sync |
[production] |
| 12:43 |
<elukey@cumin1003> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1013.eqiad.wmnet with reason: host reimage |
[production] |
| 12:42 |
<ladsgroup@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2224 (T402925)', diff saved to https://phabricator.wikimedia.org/P82012 and previous config saved to /var/cache/conftool/dbconfig/20250828-124216-ladsgroup.json |
[production] |
| 12:41 |
<kharlan@deploy1003> |
kharlan: Backport for [[gerrit:1182824|hCaptcha: Enable processing of the risk score (T379179)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 12:37 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1013.eqiad.wmnet with reason: host reimage |
[production] |
| 12:35 |
<kharlan@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1182824|hCaptcha: Enable processing of the risk score (T379179)]] |
[production] |
| 12:26 |
<ladsgroup@cumin1003> |
dbctl commit (dc=all): 'Depooling db2224 (T402925)', diff saved to https://phabricator.wikimedia.org/P82011 and previous config saved to /var/cache/conftool/dbconfig/20250828-122608-ladsgroup.json |
[production] |
| 12:26 |
<ladsgroup@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2224.codfw.wmnet with reason: Maintenance |
[production] |
| 12:25 |
<ladsgroup@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2217 (T402925)', diff saved to https://phabricator.wikimedia.org/P82010 and previous config saved to /var/cache/conftool/dbconfig/20250828-122545-ladsgroup.json |
[production] |
| 12:18 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2204 (T401906)', diff saved to https://phabricator.wikimedia.org/P82009 and previous config saved to /var/cache/conftool/dbconfig/20250828-121844-fceratto.json |
[production] |
| 12:17 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db2204 (T401906)', diff saved to https://phabricator.wikimedia.org/P82008 and previous config saved to /var/cache/conftool/dbconfig/20250828-121720-fceratto.json |
[production] |
| 12:17 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2204.codfw.wmnet with reason: Maintenance |
[production] |
| 12:16 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2197.codfw.wmnet with reason: Maintenance |
[production] |
| 12:16 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2189 (T401906)', diff saved to https://phabricator.wikimedia.org/P82007 and previous config saved to /var/cache/conftool/dbconfig/20250828-121634-fceratto.json |
[production] |
| 12:14 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.reimage for host ml-serve1013.eqiad.wmnet with OS bookworm |
[production] |
| 12:10 |
<ladsgroup@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P82006 and previous config saved to /var/cache/conftool/dbconfig/20250828-121038-ladsgroup.json |
[production] |
| 12:04 |
<moritzm> |
upgrading debmonitor to Envoy 1.26.8 T402584 |
[production] |
| 12:01 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P82005 and previous config saved to /var/cache/conftool/dbconfig/20250828-120128-fceratto.json |
[production] |
| 11:55 |
<ladsgroup@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P82003 and previous config saved to /var/cache/conftool/dbconfig/20250828-115530-ladsgroup.json |
[production] |
| 11:52 |
<Amir1> |
delete from recentchanges where rc_timestamp < '20250725000000'; on all.dblist (T403002) |
[production] |
| 11:46 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P82002 and previous config saved to /var/cache/conftool/dbconfig/20250828-114620-fceratto.json |
[production] |
| 11:40 |
<ladsgroup@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2217 (T402925)', diff saved to https://phabricator.wikimedia.org/P82001 and previous config saved to /var/cache/conftool/dbconfig/20250828-114023-ladsgroup.json |
[production] |
| 11:31 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2189 (T401906)', diff saved to https://phabricator.wikimedia.org/P82000 and previous config saved to /var/cache/conftool/dbconfig/20250828-113113-fceratto.json |
[production] |
| 11:28 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db2189 (T401906)', diff saved to https://phabricator.wikimedia.org/P81999 and previous config saved to /var/cache/conftool/dbconfig/20250828-112847-fceratto.json |
[production] |
| 11:28 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2189.codfw.wmnet with reason: Maintenance |
[production] |
| 11:28 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2175 (T401906)', diff saved to https://phabricator.wikimedia.org/P81998 and previous config saved to /var/cache/conftool/dbconfig/20250828-112824-fceratto.json |
[production] |
| 11:24 |
<ladsgroup@cumin1003> |
dbctl commit (dc=all): 'Depooling db2217 (T402925)', diff saved to https://phabricator.wikimedia.org/P81997 and previous config saved to /var/cache/conftool/dbconfig/20250828-112403-ladsgroup.json |
[production] |
| 11:23 |
<ladsgroup@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2217.codfw.wmnet with reason: Maintenance |
[production] |
| 11:23 |
<ladsgroup@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2214 (T402925)', diff saved to https://phabricator.wikimedia.org/P81996 and previous config saved to /var/cache/conftool/dbconfig/20250828-112340-ladsgroup.json |
[production] |
| 11:13 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P81995 and previous config saved to /var/cache/conftool/dbconfig/20250828-111317-fceratto.json |
[production] |
| 11:08 |
<ladsgroup@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P81994 and previous config saved to /var/cache/conftool/dbconfig/20250828-110833-ladsgroup.json |
[production] |
| 10:58 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P81993 and previous config saved to /var/cache/conftool/dbconfig/20250828-105810-fceratto.json |
[production] |
| 10:53 |
<ladsgroup@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P81992 and previous config saved to /var/cache/conftool/dbconfig/20250828-105326-ladsgroup.json |
[production] |