5751-5800 of 10000 results (62ms)
2022-02-02 ยง
14:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P19959 and previous config saved to /var/cache/conftool/dbconfig/20220202-141709-marostegui.json [production]
14:16 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
14:15 <XioNoX> cr2-eqdfw: push Capirca generated loopback filters [production]
14:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove weight from es1020 - as it is the master', diff saved to https://phabricator.wikimedia.org/P19958 and previous config saved to /var/cache/conftool/dbconfig/20220202-141455-marostegui.json [production]
14:13 <vgutierrez> pool cp1087 running envoy as TLS terminator - T271421 [production]
14:09 <XioNoX> cr2-eqord: push Capirca generated loopback filters [production]
14:08 <marostegui@cumin1001> dbctl commit (dc=all): 'db1179 (re)pooling @ 10%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P19957 and previous config saved to /var/cache/conftool/dbconfig/20220202-140818-root.json [production]
14:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1179 schema change', diff saved to https://phabricator.wikimedia.org/P19956 and previous config saved to /var/cache/conftool/dbconfig/20220202-140317-marostegui.json [production]
14:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 100%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P19955 and previous config saved to /var/cache/conftool/dbconfig/20220202-140239-root.json [production]
14:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P19954 and previous config saved to /var/cache/conftool/dbconfig/20220202-140204-marostegui.json [production]
13:50 <elukey> move docker on ml-serve-ctrl* nodes from device mapper to overlay2 [production]
13:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 75%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P19953 and previous config saved to /var/cache/conftool/dbconfig/20220202-134735-root.json [production]
13:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T300402)', diff saved to https://phabricator.wikimedia.org/P19952 and previous config saved to /var/cache/conftool/dbconfig/20220202-134659-marostegui.json [production]
13:40 <XioNoX> ULSFO routers: push Capirca generated loopback filters [production]
13:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1101:3317 (T300402)', diff saved to https://phabricator.wikimedia.org/P19951 and previous config saved to /var/cache/conftool/dbconfig/20220202-133713-marostegui.json [production]
13:37 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance [production]
13:37 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance [production]
13:35 <otto@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync on production [production]
13:34 <otto@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync on canary [production]
13:34 <otto@deploy1002> helmfile [eqiad] START helmfile.d/services/eventgate-main: sync on production [production]
13:34 <otto@deploy1002> helmfile [eqiad] START helmfile.d/services/eventgate-main: sync on canary [production]
13:33 <otto@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync on canary [production]
13:33 <otto@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync on production [production]
13:32 <otto@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-main: sync on production [production]
13:32 <otto@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-main: sync on canary [production]
13:32 <ottomata> roll restarting eventgate-main to pick up stream-configs for rdf-streaming-updater.reconcile [production]
13:32 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 50%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P19949 and previous config saved to /var/cache/conftool/dbconfig/20220202-133231-root.json [production]
13:31 <otto@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync on canary [production]
13:31 <otto@deploy1002> helmfile [eqiad] START helmfile.d/services/eventgate-main: sync on canary [production]
13:31 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
13:30 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance [production]
13:30 <aqu@deploy1002> Finished deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) (duration: 00m 08s) [production]
13:30 <aqu@deploy1002> Started deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) [production]
13:29 <otto@deploy1002> helmfile [staging] DONE helmfile.d/services/eventgate-main: sync on production [production]
13:28 <otto@deploy1002> helmfile [staging] DONE helmfile.d/services/eventgate-main: sync on canary [production]
13:28 <otto@deploy1002> helmfile [staging] START helmfile.d/services/eventgate-main: sync on production [production]
13:25 <XioNoX> rename cr3-ulsfo loopback terms in preparation of move to Capirca [production]
13:25 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
13:25 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance [production]
13:25 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T300402)', diff saved to https://phabricator.wikimedia.org/P19947 and previous config saved to /var/cache/conftool/dbconfig/20220202-132510-marostegui.json [production]
13:17 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 25%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P19946 and previous config saved to /var/cache/conftool/dbconfig/20220202-131728-root.json [production]
13:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P19945 and previous config saved to /var/cache/conftool/dbconfig/20220202-131006-marostegui.json [production]
13:05 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
13:04 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
13:04 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
13:03 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
13:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1166 (re)pooling @ 10%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P19944 and previous config saved to /var/cache/conftool/dbconfig/20220202-130224-root.json [production]
12:59 <taavi@deploy1002> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:670224|ULS: Remove unused ULSEventLogging variable (T275894)]] (duration: 00m 49s) [production]
12:58 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
12:57 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]