|
2023-11-21
ยง
|
| 10:00 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.migrate-host for host gitlab-runner1002.eqiad.wmnet |
[production] |
| 10:00 |
<hnowlan@deploy1002> |
helmfile [staging] START helmfile.d/services/api-gateway: apply |
[production] |
| 09:53 |
<oblivian@deploy2002> |
helmfile [staging] DONE helmfile.d/services/mobileapps: apply |
[production] |
| 09:51 |
<oblivian@deploy2002> |
helmfile [staging] START helmfile.d/services/mobileapps: apply |
[production] |
| 09:51 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2180 (re)pooling @ 90%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53667 and previous config saved to /var/cache/conftool/dbconfig/20231121-095102-arnaudb.json |
[production] |
| 09:50 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 90%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53666 and previous config saved to /var/cache/conftool/dbconfig/20231121-095031-arnaudb.json |
[production] |
| 09:35 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2180 (re)pooling @ 75%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53665 and previous config saved to /var/cache/conftool/dbconfig/20231121-093557-arnaudb.json |
[production] |
| 09:35 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 75%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53664 and previous config saved to /var/cache/conftool/dbconfig/20231121-093526-arnaudb.json |
[production] |
| 09:24 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs[2011-2013].codfw.wmnet} and A:lvs (T351069) |
[production] |
| 09:20 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2180 (re)pooling @ 60%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53663 and previous config saved to /var/cache/conftool/dbconfig/20231121-092052-arnaudb.json |
[production] |
| 09:20 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 60%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53662 and previous config saved to /var/cache/conftool/dbconfig/20231121-092021-arnaudb.json |
[production] |
| 09:19 |
<vgutierrez@cumin1001> |
START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs[2011-2013].codfw.wmnet} and A:lvs (T351069) |
[production] |
| 09:18 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs2014.codfw.wmnet} and A:lvs (T351069) |
[production] |
| 09:18 |
<vgutierrez@cumin1001> |
START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs2014.codfw.wmnet} and A:lvs (T351069) |
[production] |
| 09:17 |
<vgutierrez> |
updating pybal to 1.5.14 on codfw - T351069 |
[production] |
| 09:17 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs[5004-5005].eqsin.wmnet} and A:lvs (T351069) |
[production] |
| 09:16 |
<vgutierrez@cumin1001> |
START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs[5004-5005].eqsin.wmnet} and A:lvs (T351069) |
[production] |
| 09:15 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs5006.eqsin.wmnet} and A:lvs (T351069) |
[production] |
| 09:15 |
<vgutierrez@cumin1001> |
START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs5006.eqsin.wmnet} and A:lvs (T351069) |
[production] |
| 09:14 |
<vgutierrez> |
updating pybal to 1.5.14 on eqsin - T351069 |
[production] |
| 09:12 |
<vgutierrez@cumin1001> |
END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs[4008-4009].ulsfo.wmnet} and A:lvs (T351069) |
[production] |
| 09:10 |
<vgutierrez> |
updating pybal to 1.5.14 on ulsfo - T351069 |
[production] |
| 09:09 |
<vgutierrez@cumin1001> |
START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs[4008-4009].ulsfo.wmnet} and A:lvs (T351069) |
[production] |
| 09:05 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2180 (re)pooling @ 45%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53661 and previous config saved to /var/cache/conftool/dbconfig/20231121-090547-arnaudb.json |
[production] |
| 09:05 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 45%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53660 and previous config saved to /var/cache/conftool/dbconfig/20231121-090516-arnaudb.json |
[production] |
| 08:50 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2180 (re)pooling @ 30%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53659 and previous config saved to /var/cache/conftool/dbconfig/20231121-085042-arnaudb.json |
[production] |
| 08:50 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 30%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53658 and previous config saved to /var/cache/conftool/dbconfig/20231121-085011-arnaudb.json |
[production] |
| 08:41 |
<vgutierrez> |
updating pybal to 1.5.14 on lvs4010 - T351069 |
[production] |
| 08:38 |
<awight> |
scap window cancelled due to k8s error |
[production] |
| 08:37 |
<awight@deploy2002> |
Finished scap: Backport for [[gerrit:976151|Revert "Enable Reference Previews on all wikis"]] (duration: 07m 08s) |
[production] |
| 08:37 |
<vgutierrez> |
upload pybal 1.15.14 to apt.wm.o (bullseye-wikimedia) - T348837 |
[production] |
| 08:35 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2180 (re)pooling @ 15%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53657 and previous config saved to /var/cache/conftool/dbconfig/20231121-083537-arnaudb.json |
[production] |
| 08:35 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 15%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53656 and previous config saved to /var/cache/conftool/dbconfig/20231121-083504-arnaudb.json |
[production] |
| 08:32 |
<awight@deploy2002> |
awight and trainbranchbot: Continuing with sync |
[production] |
| 08:31 |
<awight@deploy2002> |
awight and trainbranchbot: Backport for [[gerrit:976151|Revert "Enable Reference Previews on all wikis"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:31 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.puppet.migrate-role (exit_code=99) for role: ncredir |
[production] |
| 08:30 |
<awight@deploy2002> |
Started scap: Backport for [[gerrit:976151|Revert "Enable Reference Previews on all wikis"]] |
[production] |
| 08:28 |
<awight@deploy2002> |
Sync cancelled. |
[production] |
| 08:28 |
<awight@deploy2002> |
wmde-fisch and awight: Backport for [[gerrit:971882|Enable Reference Previews on all wikis (T282999)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:27 |
<awight@deploy2002> |
Started scap: Backport for [[gerrit:971882|Enable Reference Previews on all wikis (T282999)]] |
[production] |
| 08:24 |
<awight@deploy2002> |
Finished scap: Backport for [[gerrit:971882|Enable Reference Previews on all wikis (T282999)]] (duration: 15m 02s) |
[production] |
| 08:20 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2180 (re)pooling @ 10%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53655 and previous config saved to /var/cache/conftool/dbconfig/20231121-082032-arnaudb.json |
[production] |
| 08:20 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 10%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53654 and previous config saved to /var/cache/conftool/dbconfig/20231121-082000-arnaudb.json |
[production] |
| 08:18 |
<awight@deploy2002> |
awight and wmde-fisch: Continuing with sync |
[production] |
| 08:16 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host druid1011.eqiad.wmnet with OS bullseye |
[production] |
| 08:11 |
<awight@deploy2002> |
awight and wmde-fisch: Backport for [[gerrit:971882|Enable Reference Previews on all wikis (T282999)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 08:09 |
<awight@deploy2002> |
Started scap: Backport for [[gerrit:971882|Enable Reference Previews on all wikis (T282999)]] |
[production] |
| 08:05 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2180 (re)pooling @ 5%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53653 and previous config saved to /var/cache/conftool/dbconfig/20231121-080527-arnaudb.json |
[production] |
| 08:04 |
<arnaudb@cumin1001> |
dbctl commit (dc=all): 'db2178 (re)pooling @ 5%: Post warmup repooling', diff saved to https://phabricator.wikimedia.org/P53652 and previous config saved to /var/cache/conftool/dbconfig/20231121-080455-arnaudb.json |
[production] |
| 07:52 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on druid1011.eqiad.wmnet with reason: host reimage |
[production] |