|
2024-02-07
ยง
|
| 14:20 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance |
[production] |
| 14:18 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'es2024 (re)pooling @ 60%: kernel upgrade done', diff saved to https://phabricator.wikimedia.org/P56440 and previous config saved to /var/cache/conftool/dbconfig/20240207-141812-arnaudb.json |
[production] |
| 14:17 |
<logmsgbot> |
lucaswerkmeister-wmde@deploy2002 Finished scap: Backport for [[gerrit:997878|Fix PermissionException being logged (T356223)]], [[gerrit:997877|Fix PermissionException being logged (T356223)]] (duration: 08m 08s) |
[production] |
| 14:11 |
<logmsgbot> |
lucaswerkmeister-wmde@deploy2002 jforrester and lucaswerkmeister-wmde: Continuing with sync |
[production] |
| 14:11 |
<logmsgbot> |
lucaswerkmeister-wmde@deploy2002 jforrester and lucaswerkmeister-wmde: Backport for [[gerrit:997878|Fix PermissionException being logged (T356223)]], [[gerrit:997877|Fix PermissionException being logged (T356223)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 14:09 |
<logmsgbot> |
lucaswerkmeister-wmde@deploy2002 Started scap: Backport for [[gerrit:997878|Fix PermissionException being logged (T356223)]], [[gerrit:997877|Fix PermissionException being logged (T356223)]] |
[production] |
| 14:09 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2105 (re)pooling @ 75%: kernel upgrade done', diff saved to https://phabricator.wikimedia.org/P56439 and previous config saved to /var/cache/conftool/dbconfig/20240207-140918-arnaudb.json |
[production] |
| 14:03 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'es2024 (re)pooling @ 30%: kernel upgrade done', diff saved to https://phabricator.wikimedia.org/P56438 and previous config saved to /var/cache/conftool/dbconfig/20240207-140306-arnaudb.json |
[production] |
| 13:54 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2105 (re)pooling @ 60%: kernel upgrade done', diff saved to https://phabricator.wikimedia.org/P56437 and previous config saved to /var/cache/conftool/dbconfig/20240207-135412-arnaudb.json |
[production] |
| 13:54 |
<klausman@deploy2002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 13:53 |
<klausman@deploy2002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 13:53 |
<klausman@deploy2002> |
helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 13:52 |
<klausman@deploy2002> |
helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 13:52 |
<klausman@deploy2002> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 13:52 |
<klausman@deploy2002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 13:48 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'es2024 (re)pooling @ 15%: kernel upgrade done', diff saved to https://phabricator.wikimedia.org/P56436 and previous config saved to /var/cache/conftool/dbconfig/20240207-134801-arnaudb.json |
[production] |
| 13:39 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2105 (re)pooling @ 30%: kernel upgrade done', diff saved to https://phabricator.wikimedia.org/P56435 and previous config saved to /var/cache/conftool/dbconfig/20240207-133907-arnaudb.json |
[production] |
| 13:32 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=1) rolling reboot on A:ncredir |
[production] |
| 13:26 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'T344589 - depool es2024', diff saved to https://phabricator.wikimedia.org/P56434 and previous config saved to /var/cache/conftool/dbconfig/20240207-132559-arnaudb.json |
[production] |
| 13:25 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on es2024.codfw.wmnet with reason: T344589 - kernel upgrade |
[production] |
| 13:25 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on es2024.codfw.wmnet with reason: T344589 - kernel upgrade |
[production] |
| 13:24 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2105 (re)pooling @ 15%: kernel upgrade done', diff saved to https://phabricator.wikimedia.org/P56433 and previous config saved to /var/cache/conftool/dbconfig/20240207-132402-arnaudb.json |
[production] |
| 12:46 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'T344589 - depool db2105', diff saved to https://phabricator.wikimedia.org/P56432 and previous config saved to /var/cache/conftool/dbconfig/20240207-124605-arnaudb.json |
[production] |
| 12:45 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2105.codfw.wmnet with reason: T344589 - kernel upgrade |
[production] |
| 12:45 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on db2105.codfw.wmnet with reason: T344589 - kernel upgrade |
[production] |
| 12:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2190 (T355609)', diff saved to https://phabricator.wikimedia.org/P56431 and previous config saved to /var/cache/conftool/dbconfig/20240207-124409-marostegui.json |
[production] |
| 12:35 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1019*,lvs2013*} and A:lvs (T334488) |
[production] |
| 12:34 |
<hnowlan@cumin2002> |
START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1019*,lvs2013*} and A:lvs (T334488) |
[production] |
| 12:33 |
<jmm@cumin2002> |
START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling reboot on A:ncredir |
[production] |
| 12:32 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1020*,lvs2014*} and A:lvs (T334488) |
[production] |
| 12:31 |
<hnowlan@cumin2002> |
START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1020*,lvs2014*} and A:lvs (T334488) |
[production] |
| 12:31 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1016.eqiad.wmnet |
[production] |
| 12:29 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P56430 and previous config saved to /var/cache/conftool/dbconfig/20240207-122903-marostegui.json |
[production] |
| 12:25 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host lvs1016.eqiad.wmnet |
[production] |
| 12:25 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1015.eqiad.wmnet |
[production] |
| 12:19 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host lvs1015.eqiad.wmnet |
[production] |
| 12:18 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1014.eqiad.wmnet |
[production] |
| 12:17 |
<claime> |
trafficserver: move 40% of traffic to mw on k8s - T355532 |
[production] |
| 12:14 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mw-web: apply |
[production] |
| 12:14 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/mw-web: apply |
[production] |
| 12:14 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/mw-web: apply |
[production] |
| 12:14 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host lvs1014.eqiad.wmnet |
[production] |
| 12:14 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/mw-web: apply |
[production] |
| 12:13 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P56429 and previous config saved to /var/cache/conftool/dbconfig/20240207-121356-marostegui.json |
[production] |
| 12:13 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply |
[production] |
| 12:12 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/mw-api-ext: apply |
[production] |
| 12:12 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply |
[production] |
| 12:12 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply |
[production] |
| 12:12 |
<claime> |
mw-web, mw-api-ext: Raise replicas for 40% traffic - T355532 |
[production] |
| 12:02 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1013.eqiad.wmnet |
[production] |