|
2023-05-11
ยง
|
| 14:08 |
<bking@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 14:08 |
<thcipriani> |
starting Gerrit Switchover (Take II): The Reckoning |
[production] |
| 14:05 |
<gmodena@deploy1002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply |
[production] |
| 14:05 |
<gmodena@deploy1002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply |
[production] |
| 14:04 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance es2024 (T335845)', diff saved to https://phabricator.wikimedia.org/P48192 and previous config saved to /var/cache/conftool/dbconfig/20230511-140440-ladsgroup.json |
[production] |
| 13:57 |
<elukey> |
upgrade benthos (4.9.1 -> 4.15.0) on centrallog nodes - T331801 |
[production] |
| 13:53 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling es2024 (T335845)', diff saved to https://phabricator.wikimedia.org/P48191 and previous config saved to /var/cache/conftool/dbconfig/20230511-135335-ladsgroup.json |
[production] |
| 13:53 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2024.codfw.wmnet with reason: Maintenance |
[production] |
| 13:53 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es2024.codfw.wmnet with reason: Maintenance |
[production] |
| 13:49 |
<moritzm> |
uploaded wmf-laptop 0.5.7 to component/wmf-sre-laptop |
[production] |
| 13:41 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2002.codfw.wmnet with OS bullseye |
[production] |
| 13:38 |
<gmodena@deploy1002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply |
[production] |
| 13:38 |
<gmodena@deploy1002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply |
[production] |
| 13:29 |
<slyngshede@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2002.codfw.wmnet with reason: host reimage |
[production] |
| 13:26 |
<gmodena@deploy1002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply |
[production] |
| 13:26 |
<gmodena@deploy1002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply |
[production] |
| 13:26 |
<slyngshede@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2002.codfw.wmnet with reason: host reimage |
[production] |
| 13:23 |
<gmodena@deploy1002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply |
[production] |
| 13:23 |
<gmodena@deploy1002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply |
[production] |
| 13:21 |
<elukey> |
upload benthos 4.15.0-1 to {buster,bullseye}-wikimedia - T331801 |
[production] |
| 13:13 |
<slyngshede@cumin1001> |
START - Cookbook sre.hosts.reimage for host testvm2002.codfw.wmnet with OS bullseye |
[production] |
| 13:07 |
<filippo@cumin1001> |
conftool action : set/weight=100; selector: name=thanos-fe2004.codfw.wmnet |
[production] |
| 13:07 |
<filippo@cumin1001> |
conftool action : set/pooled=no; selector: name=thanos-fe2004.codfw.wmnet,service=thanos-web |
[production] |
| 13:07 |
<filippo@cumin1001> |
conftool action : set/pooled=yes; selector: name=thanos-fe2004.codfw.wmnet |
[production] |
| 13:07 |
<filippo@cumin1001> |
conftool action : set/pooled=true; selector: name=thanos-fe2004.eqiad.wmnet |
[production] |
| 13:06 |
<filippo@cumin1001> |
conftool action : set/pooled=yes; selector: name=thanos-fe2004.eqiad.wmnet |
[production] |
| 13:06 |
<filippo@cumin1001> |
conftool action : set/weight=100; selector: name=thanos-fe2004.eqiad.wmnet |
[production] |
| 13:05 |
<filippo@cumin1001> |
conftool action : set/pooled=no; selector: name=thanos-fe1004.eqiad.wmnet,service=thanos-web |
[production] |
| 13:05 |
<filippo@cumin1001> |
conftool action : set/weight=100; selector: name=thanos-fe1004.eqiad.wmnet |
[production] |
| 12:58 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:919051|Add outreachwiki to wikidataclient dblist (T171140)]] (duration: 11m 05s) |
[production] |
| 12:54 |
<godog> |
roll-restart thanos-fe swift-proxy to apply config changes - T336348 |
[production] |
| 12:48 |
<ladsgroup@deploy1002> |
ladsgroup: Backport for [[gerrit:919051|Add outreachwiki to wikidataclient dblist (T171140)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 12:47 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:919051|Add outreachwiki to wikidataclient dblist (T171140)]] |
[production] |
| 12:41 |
<Amir1> |
creating wikidata client tables for outreachwiki (T171140) |
[production] |
| 12:18 |
<eoghan@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab2002.wikimedia.org with OS bullseye |
[production] |
| 12:01 |
<jelto@cumin1001> |
END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1004.wikimedia.org with reason: Install software version upgrade |
[production] |
| 11:57 |
<eoghan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab2002.wikimedia.org with reason: host reimage |
[production] |
| 11:54 |
<eoghan@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab2002.wikimedia.org with reason: host reimage |
[production] |
| 11:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1137 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48190 and previous config saved to /var/cache/conftool/dbconfig/20230511-115201-root.json |
[production] |
| 11:39 |
<eoghan@cumin1001> |
START - Cookbook sre.hosts.reimage for host gitlab2002.wikimedia.org with OS bullseye |
[production] |
| 11:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1137 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48189 and previous config saved to /var/cache/conftool/dbconfig/20230511-113657-root.json |
[production] |
| 11:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1137 (re)pooling @ 50%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48187 and previous config saved to /var/cache/conftool/dbconfig/20230511-112152-root.json |
[production] |
| 11:08 |
<jelto@cumin1001> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Install software version upgrade |
[production] |
| 11:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1137 (re)pooling @ 25%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48186 and previous config saved to /var/cache/conftool/dbconfig/20230511-110647-root.json |
[production] |
| 10:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1137 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48185 and previous config saved to /var/cache/conftool/dbconfig/20230511-105142-root.json |
[production] |
| 10:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1137 (re)pooling @ 5%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48184 and previous config saved to /var/cache/conftool/dbconfig/20230511-103638-root.json |
[production] |
| 10:23 |
<hnowlan@deploy1002> |
helmfile [staging] DONE helmfile.d/services/thumbor: apply |
[production] |
| 10:23 |
<hnowlan@deploy1002> |
helmfile [staging] START helmfile.d/services/thumbor: apply |
[production] |
| 10:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1137 (re)pooling @ 3%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48183 and previous config saved to /var/cache/conftool/dbconfig/20230511-102133-root.json |
[production] |
| 10:17 |
<moritzm> |
installing modsecurity-crs security updates |
[production] |