101-150 of 10000 results (67ms)
2024-02-19 ยง
17:38 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: recloning db2156 (T352010) [production]
17:34 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2194:3317 (re)pooling @ 40%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57126 and previous config saved to /var/cache/conftool/dbconfig/20240219-173438-arnaudb.json [production]
17:34 <ladsgroup@cumin1002> dbctl commit (dc=all): 'db2149 for maint', diff saved to https://phabricator.wikimedia.org/P57125 and previous config saved to /var/cache/conftool/dbconfig/20240219-173411-ladsgroup.json [production]
17:24 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2114', diff saved to https://phabricator.wikimedia.org/P57124 and previous config saved to /var/cache/conftool/dbconfig/20240219-172412-arnaudb.json [production]
17:19 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2194:3317 (re)pooling @ 30%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57123 and previous config saved to /var/cache/conftool/dbconfig/20240219-171933-arnaudb.json [production]
17:09 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2114', diff saved to https://phabricator.wikimedia.org/P57122 and previous config saved to /var/cache/conftool/dbconfig/20240219-170906-arnaudb.json [production]
17:04 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2194:3317 (re)pooling @ 20%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57121 and previous config saved to /var/cache/conftool/dbconfig/20240219-170428-arnaudb.json [production]
16:55 <marostegui@cumin1002> dbctl commit (dc=all): 'db1213 (re)pooling @ 100%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57120 and previous config saved to /var/cache/conftool/dbconfig/20240219-165503-root.json [production]
16:54 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2114 (T357189)', diff saved to https://phabricator.wikimedia.org/P57119 and previous config saved to /var/cache/conftool/dbconfig/20240219-165400-arnaudb.json [production]
16:50 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2114 (T357189)', diff saved to https://phabricator.wikimedia.org/P57118 and previous config saved to /var/cache/conftool/dbconfig/20240219-165032-arnaudb.json [production]
16:50 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
16:50 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
16:49 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2194:3317 (re)pooling @ 10%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57117 and previous config saved to /var/cache/conftool/dbconfig/20240219-164924-arnaudb.json [production]
16:48 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance [production]
16:48 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance [production]
16:48 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1231 (T357189)', diff saved to https://phabricator.wikimedia.org/P57116 and previous config saved to /var/cache/conftool/dbconfig/20240219-164809-arnaudb.json [production]
16:39 <marostegui@cumin1002> dbctl commit (dc=all): 'db1213 (re)pooling @ 75%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57115 and previous config saved to /var/cache/conftool/dbconfig/20240219-163958-root.json [production]
16:38 <jgiannelos@deploy2002> Finished deploy [restbase/deploy@7e5e720]: Disable parsoid storage on restbase[1031:1033] (duration: 01m 55s) [production]
16:36 <jgiannelos@deploy2002> Started deploy [restbase/deploy@7e5e720]: Disable parsoid storage on restbase[1031:1033] [production]
16:35 <jgiannelos@deploy2002> Finished deploy [restbase/deploy@7e5e720]: Disable parsoid storage on restbase[2033:2035] (duration: 01m 19s) [production]
16:34 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2194:3317 (re)pooling @ 8%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57114 and previous config saved to /var/cache/conftool/dbconfig/20240219-163419-arnaudb.json [production]
16:33 <jgiannelos@deploy2002> Started deploy [restbase/deploy@7e5e720]: Disable parsoid storage on restbase[2033:2035] [production]
16:33 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P57113 and previous config saved to /var/cache/conftool/dbconfig/20240219-163303-arnaudb.json [production]
16:32 <jgiannelos@deploy2002> deploy aborted: Disable parsoid storage on all nodes (duration: 01m 57s) [production]
16:30 <jgiannelos@deploy2002> Started deploy [restbase/deploy@7e5e720]: Disable parsoid storage on all nodes [production]
16:30 <jgiannelos@deploy2002> Finished deploy [restbase/deploy@7e5e720]: Disable parsoid storage on all nodes (duration: 00m 07s) [production]
16:30 <hnowlan@deploy2002> helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
16:30 <hnowlan@deploy2002> helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync [production]
16:30 <jgiannelos@deploy2002> Started deploy [restbase/deploy@7e5e720]: Disable parsoid storage on all nodes [production]
16:29 <jgiannelos@deploy2002> deploy aborted: Disable parsoid storage on all nodes (duration: 00m 08s) [production]
16:29 <jgiannelos@deploy2002> Started deploy [restbase/deploy@7e5e720]: Disable parsoid storage on all nodes [production]
16:29 <hnowlan@deploy2002> helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
16:29 <hnowlan@deploy2002> helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync [production]
16:24 <marostegui@cumin1002> dbctl commit (dc=all): 'db1213 (re)pooling @ 50%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57112 and previous config saved to /var/cache/conftool/dbconfig/20240219-162453-root.json [production]
16:21 <jgiannelos@deploy2002> Finished deploy [restbase/deploy@e5ed8d0]: (no justification provided) (duration: 00m 04s) [production]
16:21 <jgiannelos@deploy2002> Started deploy [restbase/deploy@e5ed8d0]: (no justification provided) [production]
16:19 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2194:3317 (re)pooling @ 4%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57111 and previous config saved to /var/cache/conftool/dbconfig/20240219-161914-arnaudb.json [production]
16:19 <jgiannelos@deploy2002> Finished deploy [restbase/deploy@e5ed8d0]: (no justification provided) (duration: 00m 07s) [production]
16:18 <jgiannelos@deploy2002> Started deploy [restbase/deploy@e5ed8d0]: (no justification provided) [production]
16:17 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P57110 and previous config saved to /var/cache/conftool/dbconfig/20240219-161756-arnaudb.json [production]
16:17 <jgiannelos@deploy2002> deploy aborted: Deploy latest restbase config in all nodes (duration: 00m 04s) [production]
16:16 <jgiannelos@deploy2002> Started deploy [restbase/deploy@e5ed8d0]: Deploy latest restbase config in all nodes [production]
16:14 <jgiannelos@deploy2002> Finished deploy [restbase/deploy@e5ed8d0]: (no justification provided) (duration: 00m 08s) [production]
16:14 <jgiannelos@deploy2002> Started deploy [restbase/deploy@e5ed8d0]: (no justification provided) [production]
16:09 <marostegui@cumin1002> dbctl commit (dc=all): 'db1213 (re)pooling @ 25%: After migration to 10.6', diff saved to https://phabricator.wikimedia.org/P57109 and previous config saved to /var/cache/conftool/dbconfig/20240219-160948-root.json [production]
16:04 <jgiannelos@deploy2002> Finished deploy [restbase/deploy@e5ed8d0]: (no justification provided) (duration: 00m 23s) [production]
16:04 <jgiannelos@deploy2002> Started deploy [restbase/deploy@e5ed8d0]: (no justification provided) [production]
16:04 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2194:3317 (re)pooling @ 2%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57108 and previous config saved to /var/cache/conftool/dbconfig/20240219-160409-arnaudb.json [production]
16:02 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1231 (T357189)', diff saved to https://phabricator.wikimedia.org/P57107 and previous config saved to /var/cache/conftool/dbconfig/20240219-160249-arnaudb.json [production]
16:02 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2169:3316 (re)pooling @ 100%: Cloning to db2194 done', diff saved to https://phabricator.wikimedia.org/P57106 and previous config saved to /var/cache/conftool/dbconfig/20240219-160221-arnaudb.json [production]