|
2022-02-02
ยง
|
| 14:32 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T300402)', diff saved to https://phabricator.wikimedia.org/P19961 and previous config saved to /var/cache/conftool/dbconfig/20220202-143214-marostegui.json |
[production] |
| 14:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 25%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P19960 and previous config saved to /var/cache/conftool/dbconfig/20220202-142321-root.json |
[production] |
| 14:21 |
<XioNoX> |
eqsin: push Capirca generated loopback filters |
[production] |
| 14:19 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 14:18 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 14:18 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |
| 14:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P19959 and previous config saved to /var/cache/conftool/dbconfig/20220202-141709-marostegui.json |
[production] |
| 14:16 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn |
[production] |
| 14:15 |
<XioNoX> |
cr2-eqdfw: push Capirca generated loopback filters |
[production] |
| 14:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove weight from es1020 - as it is the master', diff saved to https://phabricator.wikimedia.org/P19958 and previous config saved to /var/cache/conftool/dbconfig/20220202-141455-marostegui.json |
[production] |
| 14:13 |
<vgutierrez> |
pool cp1087 running envoy as TLS terminator - T271421 |
[production] |
| 14:09 |
<XioNoX> |
cr2-eqord: push Capirca generated loopback filters |
[production] |
| 14:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1179 (re)pooling @ 10%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P19957 and previous config saved to /var/cache/conftool/dbconfig/20220202-140818-root.json |
[production] |
| 14:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1179 schema change', diff saved to https://phabricator.wikimedia.org/P19956 and previous config saved to /var/cache/conftool/dbconfig/20220202-140317-marostegui.json |
[production] |
| 14:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1166 (re)pooling @ 100%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P19955 and previous config saved to /var/cache/conftool/dbconfig/20220202-140239-root.json |
[production] |
| 14:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P19954 and previous config saved to /var/cache/conftool/dbconfig/20220202-140204-marostegui.json |
[production] |
| 13:50 |
<elukey> |
move docker on ml-serve-ctrl* nodes from device mapper to overlay2 |
[production] |
| 13:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1166 (re)pooling @ 75%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P19953 and previous config saved to /var/cache/conftool/dbconfig/20220202-134735-root.json |
[production] |
| 13:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T300402)', diff saved to https://phabricator.wikimedia.org/P19952 and previous config saved to /var/cache/conftool/dbconfig/20220202-134659-marostegui.json |
[production] |
| 13:40 |
<XioNoX> |
ULSFO routers: push Capirca generated loopback filters |
[production] |
| 13:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1101:3317 (T300402)', diff saved to https://phabricator.wikimedia.org/P19951 and previous config saved to /var/cache/conftool/dbconfig/20220202-133713-marostegui.json |
[production] |
| 13:37 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:37 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:35 |
<otto@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync on production |
[production] |
| 13:34 |
<otto@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync on canary |
[production] |
| 13:34 |
<otto@deploy1002> |
helmfile [eqiad] START helmfile.d/services/eventgate-main: sync on production |
[production] |
| 13:34 |
<otto@deploy1002> |
helmfile [eqiad] START helmfile.d/services/eventgate-main: sync on canary |
[production] |
| 13:33 |
<otto@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync on canary |
[production] |
| 13:33 |
<otto@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync on production |
[production] |
| 13:32 |
<otto@deploy1002> |
helmfile [codfw] START helmfile.d/services/eventgate-main: sync on production |
[production] |
| 13:32 |
<otto@deploy1002> |
helmfile [codfw] START helmfile.d/services/eventgate-main: sync on canary |
[production] |
| 13:32 |
<ottomata> |
roll restarting eventgate-main to pick up stream-configs for rdf-streaming-updater.reconcile |
[production] |
| 13:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1166 (re)pooling @ 50%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P19949 and previous config saved to /var/cache/conftool/dbconfig/20220202-133231-root.json |
[production] |
| 13:31 |
<otto@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync on canary |
[production] |
| 13:31 |
<otto@deploy1002> |
helmfile [eqiad] START helmfile.d/services/eventgate-main: sync on canary |
[production] |
| 13:31 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:30 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:30 |
<aqu@deploy1002> |
Finished deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) (duration: 00m 08s) |
[production] |
| 13:30 |
<aqu@deploy1002> |
Started deploy [airflow-dags/analytics-test@3ad07a0]: (no justification provided) |
[production] |
| 13:29 |
<otto@deploy1002> |
helmfile [staging] DONE helmfile.d/services/eventgate-main: sync on production |
[production] |
| 13:28 |
<otto@deploy1002> |
helmfile [staging] DONE helmfile.d/services/eventgate-main: sync on canary |
[production] |
| 13:28 |
<otto@deploy1002> |
helmfile [staging] START helmfile.d/services/eventgate-main: sync on production |
[production] |
| 13:25 |
<XioNoX> |
rename cr3-ulsfo loopback terms in preparation of move to Capirca |
[production] |
| 13:25 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:25 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T300402)', diff saved to https://phabricator.wikimedia.org/P19947 and previous config saved to /var/cache/conftool/dbconfig/20220202-132510-marostegui.json |
[production] |
| 13:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1166 (re)pooling @ 25%: repooling after schema change', diff saved to https://phabricator.wikimedia.org/P19946 and previous config saved to /var/cache/conftool/dbconfig/20220202-131728-root.json |
[production] |
| 13:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P19945 and previous config saved to /var/cache/conftool/dbconfig/20220202-131006-marostegui.json |
[production] |
| 13:05 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn |
[production] |