351-400 of 10000 results (99ms)
2026-02-03 §
02:36 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P88437 and previous config saved to /var/cache/conftool/dbconfig/20260203-023627-marostegui.json [production]
02:21 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2227 (T415786)', diff saved to https://phabricator.wikimedia.org/P88436 and previous config saved to /var/cache/conftool/dbconfig/20260203-022119-marostegui.json [production]
02:13 <mwpresync@deploy2002> Finished scap build-images: Publishing wmf/next image (duration: 12m 39s) [production]
02:05 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
02:00 <mwpresync@deploy2002> Started scap build-images: Publishing wmf/next image [production]
00:15 <marostegui@cumin1003> dbctl commit (dc=all): 'Depooling db2227 (T415786)', diff saved to https://phabricator.wikimedia.org/P88435 and previous config saved to /var/cache/conftool/dbconfig/20260203-001511-marostegui.json [production]
00:15 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2227.codfw.wmnet with reason: Maintenance [production]
00:14 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2209 (T415786)', diff saved to https://phabricator.wikimedia.org/P88434 and previous config saved to /var/cache/conftool/dbconfig/20260203-001445-marostegui.json [production]
00:04 <robh> eqsin cp5022 troubleshooting onsite in progress [production]
2026-02-02 §
23:59 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P88433 and previous config saved to /var/cache/conftool/dbconfig/20260202-235937-marostegui.json [production]
23:44 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P88432 and previous config saved to /var/cache/conftool/dbconfig/20260202-234429-marostegui.json [production]
23:29 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2209 (T415786)', diff saved to https://phabricator.wikimedia.org/P88431 and previous config saved to /var/cache/conftool/dbconfig/20260202-232921-marostegui.json [production]
22:40 <herron> added 500G to the lv on mwlog1002 [production]
22:24 <inflatador> bking@apt1002 `sudo -E reprepro -C thirdparty/opensearch3 copy trixie-wikimedia bookworm-wikimedia opensearch` [production]
22:19 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1240.eqiad.wmnet with reason: Maintenance [production]
22:19 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1212 (T415786)', diff saved to https://phabricator.wikimedia.org/P88430 and previous config saved to /var/cache/conftool/dbconfig/20260202-221912-marostegui.json [production]
22:04 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P88429 and previous config saved to /var/cache/conftool/dbconfig/20260202-220404-marostegui.json [production]
21:48 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P88428 and previous config saved to /var/cache/conftool/dbconfig/20260202-214855-marostegui.json [production]
21:33 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1212 (T415786)', diff saved to https://phabricator.wikimedia.org/P88427 and previous config saved to /var/cache/conftool/dbconfig/20260202-213347-marostegui.json [production]
21:27 <marostegui@cumin1003> dbctl commit (dc=all): 'Depooling db2209 (T415786)', diff saved to https://phabricator.wikimedia.org/P88426 and previous config saved to /var/cache/conftool/dbconfig/20260202-212703-marostegui.json [production]
21:26 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2209.codfw.wmnet with reason: Maintenance [production]
21:26 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2194 (T415786)', diff saved to https://phabricator.wikimedia.org/P88425 and previous config saved to /var/cache/conftool/dbconfig/20260202-212638-marostegui.json [production]
21:16 <kemayo@deploy2002> Finished scap sync-world: Backport for [[gerrit:1235392|Edit check: turn off the tone a/b test on frwiki, jawiki, ptwiki (T411914)]], [[gerrit:1235111|Enable suggestions BetaFeature on beta wikis (T415504)]], [[gerrit:1230462|WikimediaCustomizations: Set WMCBadEmailDomainsFile (T397244)]], [[gerrit:1235491|filebackend: Clean up removed config params for multi-write backends (T328872)]] (duration: 10 [production]
21:12 <kemayo@deploy2002> tgr, func, kemayo, esanders: Continuing with sync [production]
21:11 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P88424 and previous config saved to /var/cache/conftool/dbconfig/20260202-211129-marostegui.json [production]
21:07 <kemayo@deploy2002> tgr, func, kemayo, esanders: Backport for [[gerrit:1235392|Edit check: turn off the tone a/b test on frwiki, jawiki, ptwiki (T411914)]], [[gerrit:1235111|Enable suggestions BetaFeature on beta wikis (T415504)]], [[gerrit:1230462|WikimediaCustomizations: Set WMCBadEmailDomainsFile (T397244)]], [[gerrit:1235491|filebackend: Clean up removed config params for multi-write backends (T328872)]] synced to [production]
21:05 <kemayo@deploy2002> Started scap sync-world: Backport for [[gerrit:1235392|Edit check: turn off the tone a/b test on frwiki, jawiki, ptwiki (T411914)]], [[gerrit:1235111|Enable suggestions BetaFeature on beta wikis (T415504)]], [[gerrit:1230462|WikimediaCustomizations: Set WMCBadEmailDomainsFile (T397244)]], [[gerrit:1235491|filebackend: Clean up removed config params for multi-write backends (T328872)]] [production]
20:56 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P88423 and previous config saved to /var/cache/conftool/dbconfig/20260202-205621-marostegui.json [production]
20:41 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2194 (T415786)', diff saved to https://phabricator.wikimedia.org/P88422 and previous config saved to /var/cache/conftool/dbconfig/20260202-204113-marostegui.json [production]
20:24 <marostegui@cumin1003> dbctl commit (dc=all): 'Depooling db1212 (T415786)', diff saved to https://phabricator.wikimedia.org/P88421 and previous config saved to /var/cache/conftool/dbconfig/20260202-202451-marostegui.json [production]
20:24 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on 6 hosts with reason: Maintenance [production]
20:24 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1212.eqiad.wmnet with reason: Maintenance [production]
20:24 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1198 (T415786)', diff saved to https://phabricator.wikimedia.org/P88420 and previous config saved to /var/cache/conftool/dbconfig/20260202-202404-marostegui.json [production]
20:08 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P88419 and previous config saved to /var/cache/conftool/dbconfig/20260202-200855-marostegui.json [production]
19:53 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P88418 and previous config saved to /var/cache/conftool/dbconfig/20260202-195345-marostegui.json [production]
19:38 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1198 (T415786)', diff saved to https://phabricator.wikimedia.org/P88417 and previous config saved to /var/cache/conftool/dbconfig/20260202-193837-marostegui.json [production]
18:42 <bvibber@deploy2002> helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply [production]
18:42 <bvibber@deploy2002> helmfile [eqiad] START helmfile.d/services/chart-renderer: apply [production]
18:41 <bvibber@deploy2002> helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply [production]
18:41 <bvibber@deploy2002> helmfile [codfw] START helmfile.d/services/chart-renderer: apply [production]
18:40 <bvibber@deploy2002> helmfile [staging] DONE helmfile.d/services/chart-renderer: apply [production]
18:40 <bvibber@deploy2002> helmfile [staging] START helmfile.d/services/chart-renderer: apply [production]
18:33 <marostegui@cumin1003> dbctl commit (dc=all): 'Depooling db2194 (T415786)', diff saved to https://phabricator.wikimedia.org/P88416 and previous config saved to /var/cache/conftool/dbconfig/20260202-183312-marostegui.json [production]
18:33 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2194.codfw.wmnet with reason: Maintenance [production]
18:32 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2190 (T415786)', diff saved to https://phabricator.wikimedia.org/P88415 and previous config saved to /var/cache/conftool/dbconfig/20260202-183248-marostegui.json [production]
18:22 <marostegui@cumin1003> dbctl commit (dc=all): 'Depooling db1198 (T415786)', diff saved to https://phabricator.wikimedia.org/P88414 and previous config saved to /var/cache/conftool/dbconfig/20260202-182210-marostegui.json [production]
18:22 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1198.eqiad.wmnet with reason: Maintenance [production]
18:21 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1189 (T415786)', diff saved to https://phabricator.wikimedia.org/P88413 and previous config saved to /var/cache/conftool/dbconfig/20260202-182144-marostegui.json [production]
18:17 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P88412 and previous config saved to /var/cache/conftool/dbconfig/20260202-181739-marostegui.json [production]
18:06 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P88411 and previous config saved to /var/cache/conftool/dbconfig/20260202-180633-marostegui.json [production]