|
2023-05-11
ยง
|
| 13:26 |
<slyngshede@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2002.codfw.wmnet with reason: host reimage |
[production] |
| 13:23 |
<gmodena@deploy1002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply |
[production] |
| 13:23 |
<gmodena@deploy1002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mediawiki-page-content-change-enrichment: apply |
[production] |
| 13:21 |
<elukey> |
upload benthos 4.15.0-1 to {buster,bullseye}-wikimedia - T331801 |
[production] |
| 13:13 |
<slyngshede@cumin1001> |
START - Cookbook sre.hosts.reimage for host testvm2002.codfw.wmnet with OS bullseye |
[production] |
| 13:07 |
<filippo@cumin1001> |
conftool action : set/weight=100; selector: name=thanos-fe2004.codfw.wmnet |
[production] |
| 13:07 |
<filippo@cumin1001> |
conftool action : set/pooled=no; selector: name=thanos-fe2004.codfw.wmnet,service=thanos-web |
[production] |
| 13:07 |
<filippo@cumin1001> |
conftool action : set/pooled=yes; selector: name=thanos-fe2004.codfw.wmnet |
[production] |
| 13:07 |
<filippo@cumin1001> |
conftool action : set/pooled=true; selector: name=thanos-fe2004.eqiad.wmnet |
[production] |
| 13:06 |
<filippo@cumin1001> |
conftool action : set/pooled=yes; selector: name=thanos-fe2004.eqiad.wmnet |
[production] |
| 13:06 |
<filippo@cumin1001> |
conftool action : set/weight=100; selector: name=thanos-fe2004.eqiad.wmnet |
[production] |
| 13:05 |
<filippo@cumin1001> |
conftool action : set/pooled=no; selector: name=thanos-fe1004.eqiad.wmnet,service=thanos-web |
[production] |
| 13:05 |
<filippo@cumin1001> |
conftool action : set/weight=100; selector: name=thanos-fe1004.eqiad.wmnet |
[production] |
| 12:58 |
<ladsgroup@deploy1002> |
Finished scap: Backport for [[gerrit:919051|Add outreachwiki to wikidataclient dblist (T171140)]] (duration: 11m 05s) |
[production] |
| 12:54 |
<godog> |
roll-restart thanos-fe swift-proxy to apply config changes - T336348 |
[production] |
| 12:48 |
<ladsgroup@deploy1002> |
ladsgroup: Backport for [[gerrit:919051|Add outreachwiki to wikidataclient dblist (T171140)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet |
[production] |
| 12:47 |
<ladsgroup@deploy1002> |
Started scap: Backport for [[gerrit:919051|Add outreachwiki to wikidataclient dblist (T171140)]] |
[production] |
| 12:41 |
<Amir1> |
creating wikidata client tables for outreachwiki (T171140) |
[production] |
| 12:18 |
<eoghan@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host gitlab2002.wikimedia.org with OS bullseye |
[production] |
| 12:01 |
<jelto@cumin1001> |
END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1004.wikimedia.org with reason: Install software version upgrade |
[production] |
| 11:57 |
<eoghan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab2002.wikimedia.org with reason: host reimage |
[production] |
| 11:54 |
<eoghan@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab2002.wikimedia.org with reason: host reimage |
[production] |
| 11:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1137 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48190 and previous config saved to /var/cache/conftool/dbconfig/20230511-115201-root.json |
[production] |
| 11:39 |
<eoghan@cumin1001> |
START - Cookbook sre.hosts.reimage for host gitlab2002.wikimedia.org with OS bullseye |
[production] |
| 11:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1137 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48189 and previous config saved to /var/cache/conftool/dbconfig/20230511-113657-root.json |
[production] |
| 11:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1137 (re)pooling @ 50%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48187 and previous config saved to /var/cache/conftool/dbconfig/20230511-112152-root.json |
[production] |
| 11:08 |
<jelto@cumin1001> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Install software version upgrade |
[production] |
| 11:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1137 (re)pooling @ 25%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48186 and previous config saved to /var/cache/conftool/dbconfig/20230511-110647-root.json |
[production] |
| 10:51 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1137 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48185 and previous config saved to /var/cache/conftool/dbconfig/20230511-105142-root.json |
[production] |
| 10:36 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1137 (re)pooling @ 5%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48184 and previous config saved to /var/cache/conftool/dbconfig/20230511-103638-root.json |
[production] |
| 10:23 |
<hnowlan@deploy1002> |
helmfile [staging] DONE helmfile.d/services/thumbor: apply |
[production] |
| 10:23 |
<hnowlan@deploy1002> |
helmfile [staging] START helmfile.d/services/thumbor: apply |
[production] |
| 10:21 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1137 (re)pooling @ 3%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48183 and previous config saved to /var/cache/conftool/dbconfig/20230511-102133-root.json |
[production] |
| 10:17 |
<moritzm> |
installing modsecurity-crs security updates |
[production] |
| 10:10 |
<moritzm> |
installing protobuf security updates |
[production] |
| 10:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1137 (re)pooling @ 1%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P48182 and previous config saved to /var/cache/conftool/dbconfig/20230511-100628-root.json |
[production] |
| 09:35 |
<moritzm> |
installing distro-info-data updates on buster |
[production] |
| 09:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1137', diff saved to https://phabricator.wikimedia.org/P48181 and previous config saved to /var/cache/conftool/dbconfig/20230511-092848-root.json |
[production] |
| 08:59 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Install software version upgrade |
[production] |
| 08:56 |
<jelto@cumin1001> |
END (ERROR) - Cookbook sre.gitlab.upgrade (exit_code=97) on GitLab host gitlab1004.wikimedia.org with reason: Install software version upgrade |
[production] |
| 08:41 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.cassandra.roll-reboot (exit_code=0) rolling reboot on A:cassandra-dev |
[production] |
| 08:40 |
<jelto@cumin1001> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Install software version upgrade |
[production] |
| 08:40 |
<elukey> |
`apt-get clean` on orespoolcounter nodes to free space in the root partition |
[production] |
| 08:33 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Install software version upgrade |
[production] |
| 08:13 |
<moritzm> |
installing Linux 4.19.282 updates on Buster systems |
[production] |
| 08:08 |
<hashar@deploy1002> |
rebuilt and synchronized wikiversions files: group2 wikis to 1.41.0-wmf.8 refs T330214 |
[production] |
| 08:06 |
<jmm@cumin2002> |
START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:cassandra-dev |
[production] |
| 08:05 |
<jelto@cumin1001> |
START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Install software version upgrade |
[production] |
| 07:43 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.cassandra.roll-reboot (exit_code=1) rolling reboot on A:cassandra-dev |
[production] |
| 07:43 |
<jmm@cumin2002> |
START - Cookbook sre.cassandra.roll-reboot rolling reboot on A:cassandra-dev |
[production] |