|
2024-02-19
ยง
|
| 18:29 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2124.codfw.wmnet with reason: Maintenance |
[production] |
| 18:29 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2117 (T357189)', diff saved to https://phabricator.wikimedia.org/P57136 and previous config saved to /var/cache/conftool/dbconfig/20240219-182905-arnaudb.json |
[production] |
| 18:19 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2194:3316 (re)pooling @ 1%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57135 and previous config saved to /var/cache/conftool/dbconfig/20240219-181958-arnaudb.json |
[production] |
| 18:19 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2194:3317 (re)pooling @ 100%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57134 and previous config saved to /var/cache/conftool/dbconfig/20240219-181953-arnaudb.json |
[production] |
| 18:14 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P57133 and previous config saved to /var/cache/conftool/dbconfig/20240219-181359-arnaudb.json |
[production] |
| 18:04 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2194:3317 (re)pooling @ 75%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57132 and previous config saved to /var/cache/conftool/dbconfig/20240219-180448-arnaudb.json |
[production] |
| 17:58 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P57131 and previous config saved to /var/cache/conftool/dbconfig/20240219-175853-arnaudb.json |
[production] |
| 17:56 |
<ladsgroup@cumin1002> |
START - Cookbook sre.mysql.clone of db2149.codfw.wmnet onto db2156.codfw.wmnet |
[production] |
| 17:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2194:3317 (re)pooling @ 50%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57130 and previous config saved to /var/cache/conftool/dbconfig/20240219-174943-arnaudb.json |
[production] |
| 17:43 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2117 (T357189)', diff saved to https://phabricator.wikimedia.org/P57129 and previous config saved to /var/cache/conftool/dbconfig/20240219-174347-arnaudb.json |
[production] |
| 17:43 |
<hnowlan> |
running `decommssion` for mw2312.codfw.wmnet,mw2313.codfw.wmnet,mw2367.codfw.wmnet,mw2369.codfw.wmnet,mw2384.codfw.wmnet,mw2385.codfw.wmnet before reimaging to k8s workers |
[production] |
| 17:39 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2117 (T357189)', diff saved to https://phabricator.wikimedia.org/P57128 and previous config saved to /var/cache/conftool/dbconfig/20240219-173941-arnaudb.json |
[production] |
| 17:39 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2117.codfw.wmnet with reason: Maintenance |
[production] |
| 17:39 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2117.codfw.wmnet with reason: Maintenance |
[production] |
| 17:39 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2114 (T357189)', diff saved to https://phabricator.wikimedia.org/P57127 and previous config saved to /var/cache/conftool/dbconfig/20240219-173919-arnaudb.json |
[production] |
| 17:38 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: recloning db2156 (T352010) |
[production] |
| 17:38 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: recloning db2156 (T352010) |
[production] |
| 17:34 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2194:3317 (re)pooling @ 40%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57126 and previous config saved to /var/cache/conftool/dbconfig/20240219-173438-arnaudb.json |
[production] |
| 17:34 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'db2149 for maint', diff saved to https://phabricator.wikimedia.org/P57125 and previous config saved to /var/cache/conftool/dbconfig/20240219-173411-ladsgroup.json |
[production] |
| 17:24 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2114', diff saved to https://phabricator.wikimedia.org/P57124 and previous config saved to /var/cache/conftool/dbconfig/20240219-172412-arnaudb.json |
[production] |
| 17:19 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2194:3317 (re)pooling @ 30%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57123 and previous config saved to /var/cache/conftool/dbconfig/20240219-171933-arnaudb.json |
[production] |
| 17:09 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2114', diff saved to https://phabricator.wikimedia.org/P57122 and previous config saved to /var/cache/conftool/dbconfig/20240219-170906-arnaudb.json |
[production] |
| 17:04 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2194:3317 (re)pooling @ 20%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57121 and previous config saved to /var/cache/conftool/dbconfig/20240219-170428-arnaudb.json |
[production] |
| 16:55 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1213 (re)pooling @ 100%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57120 and previous config saved to /var/cache/conftool/dbconfig/20240219-165503-root.json |
[production] |
| 16:54 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2114 (T357189)', diff saved to https://phabricator.wikimedia.org/P57119 and previous config saved to /var/cache/conftool/dbconfig/20240219-165400-arnaudb.json |
[production] |
| 16:50 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Depooling db2114 (T357189)', diff saved to https://phabricator.wikimedia.org/P57118 and previous config saved to /var/cache/conftool/dbconfig/20240219-165032-arnaudb.json |
[production] |
| 16:50 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: Maintenance |
[production] |
| 16:50 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: Maintenance |
[production] |
| 16:49 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2194:3317 (re)pooling @ 10%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57117 and previous config saved to /var/cache/conftool/dbconfig/20240219-164924-arnaudb.json |
[production] |
| 16:48 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:48 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:48 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1231 (T357189)', diff saved to https://phabricator.wikimedia.org/P57116 and previous config saved to /var/cache/conftool/dbconfig/20240219-164809-arnaudb.json |
[production] |
| 16:39 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1213 (re)pooling @ 75%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57115 and previous config saved to /var/cache/conftool/dbconfig/20240219-163958-root.json |
[production] |
| 16:38 |
<jgiannelos@deploy2002> |
Finished deploy [restbase/deploy@7e5e720]: Disable parsoid storage on restbase[1031:1033] (duration: 01m 55s) |
[production] |
| 16:36 |
<jgiannelos@deploy2002> |
Started deploy [restbase/deploy@7e5e720]: Disable parsoid storage on restbase[1031:1033] |
[production] |
| 16:35 |
<jgiannelos@deploy2002> |
Finished deploy [restbase/deploy@7e5e720]: Disable parsoid storage on restbase[2033:2035] (duration: 01m 19s) |
[production] |
| 16:34 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2194:3317 (re)pooling @ 8%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57114 and previous config saved to /var/cache/conftool/dbconfig/20240219-163419-arnaudb.json |
[production] |
| 16:33 |
<jgiannelos@deploy2002> |
Started deploy [restbase/deploy@7e5e720]: Disable parsoid storage on restbase[2033:2035] |
[production] |
| 16:33 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P57113 and previous config saved to /var/cache/conftool/dbconfig/20240219-163303-arnaudb.json |
[production] |
| 16:32 |
<jgiannelos@deploy2002> |
deploy aborted: Disable parsoid storage on all nodes (duration: 01m 57s) |
[production] |
| 16:30 |
<jgiannelos@deploy2002> |
Started deploy [restbase/deploy@7e5e720]: Disable parsoid storage on all nodes |
[production] |
| 16:30 |
<jgiannelos@deploy2002> |
Finished deploy [restbase/deploy@7e5e720]: Disable parsoid storage on all nodes (duration: 00m 07s) |
[production] |
| 16:30 |
<hnowlan@deploy2002> |
helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync |
[production] |
| 16:30 |
<hnowlan@deploy2002> |
helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync |
[production] |
| 16:30 |
<jgiannelos@deploy2002> |
Started deploy [restbase/deploy@7e5e720]: Disable parsoid storage on all nodes |
[production] |
| 16:29 |
<jgiannelos@deploy2002> |
deploy aborted: Disable parsoid storage on all nodes (duration: 00m 08s) |
[production] |
| 16:29 |
<jgiannelos@deploy2002> |
Started deploy [restbase/deploy@7e5e720]: Disable parsoid storage on all nodes |
[production] |
| 16:29 |
<hnowlan@deploy2002> |
helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync |
[production] |
| 16:29 |
<hnowlan@deploy2002> |
helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync |
[production] |
| 16:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1213 (re)pooling @ 50%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57112 and previous config saved to /var/cache/conftool/dbconfig/20240219-162453-root.json |
[production] |