|
2023-07-04
§
|
| 09:42 |
<cgoubert@deploy1002> |
helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:42 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:42 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:42 |
<cgoubert@deploy1002> |
helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:41 |
<cgoubert@deploy1002> |
helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:38 |
<jayme> |
updated envoyproxy to 1.23.10 on all nodes - T300324 |
[production] |
| 09:37 |
<cgoubert@deploy1002> |
helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:37 |
<cgoubert@deploy1002> |
helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:36 |
<cgoubert@deploy1002> |
helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:36 |
<cgoubert@deploy1002> |
helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:29 |
<btullis> |
restarting the yarn restart with `sudo cumin -b 5 -p 80 -s 30 A:hadoop-worker 'systemctl restart hadoop-yarn-nodemanager'` |
[analytics] |
| 09:07 |
<btullis@cumin1001> |
END (FAIL) - Cookbook sre.hadoop.roll-restart-workers (exit_code=99) restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade. |
[production] |
| 09:02 |
<jnuche@deploy1002> |
Started scap: testwikis wikis to 1.41.0-wmf.16 refs T340244 |
[production] |
| 08:57 |
<btullis> |
executing `cookbook sre.hadoop.roll-restart-workers analytics` |
[analytics] |
| 08:56 |
<btullis@cumin1001> |
START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade. |
[production] |
| 08:09 |
<wm-bot> |
<lucaswerkmeister> Double IRC messages to other bridges |
[tools.bridgebot] |
| 06:31 |
<wm-bot> |
<lokal-profil> Source code is now available at https://gist.github.com/lokal-profil/2e59b4b07d449a2c9a8ed98429787d80 |
[tools.slumpartikel] |
| 06:16 |
<wm-bot> |
<lokal-profil> Adding creator and license to source file and moving username out of source |
[tools.slumpartikel] |
|
2023-07-03
§
|
| 22:00 |
<eileen> |
civicrm upgraded from 9e04c92d to 4ca2008d |
[production] |
| 20:41 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/935166 |
[releng] |
| 20:18 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:eqiad and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) |
[production] |
| 20:15 |
<effie> |
restarting swift proxies |
[production] |
| 20:14 |
<jiji@cumin1001> |
START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:eqiad and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad) |
[production] |
| 19:11 |
<cgoubert@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 19:11 |
<cgoubert@deploy1002> |
helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 19:10 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 19:09 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 19:09 |
<cgoubert@deploy1002> |
helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 19:09 |
<cgoubert@deploy1002> |
helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 19:09 |
<wm-bot2> |
deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config.git (561b4d9) - cookbook ran by taavi@runko |
[tools] |
| 19:08 |
<wm-bot2> |
deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config.git (561b4d9) - cookbook ran by taavi@runko |
[toolsbeta] |
| 18:46 |
<wm-bot> |
<anticomposite> start webservice since it appeared to not be running for some reason |
[tools.stewardbots] |
| 16:10 |
<effie> |
restarting pybal on lvs2013 |
[production] |
| 16:04 |
<effie> |
restarting pybal on lvs2014 |
[production] |
| 15:57 |
<effie> |
restarting pybal on lvs2014 |
[production] |
| 15:52 |
<effie> |
restarting pybal on lvs1019 |
[production] |
| 15:49 |
<effie> |
restarting pybal on lvs1020 |
[production] |
| 15:34 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: name=kubestagemaster1002.eqiad.wmnet |
[production] |
| 15:34 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: name=kubestagemaster2002.codfw.wmnet |
[production] |
| 15:34 |
<jiji@cumin1001> |
conftool action : set/pooled=yes; selector: name=kubestagemaster1002.eqiad.wmnet |
[production] |
| 15:34 |
<jiji@cumin1001> |
conftool action : set/pooled=yes; selector: name=kubestagemaster2002.codfw.wmnet |
[production] |
| 15:14 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: dc=codfw,cluster=kubernetes-staging,service=kubemaster |
[production] |
| 15:12 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: name=kubestagemaster1002.eqiad.wmnet |
[production] |
| 15:12 |
<jiji@cumin1001> |
conftool action : set/weight=10; selector: name=kubestagemaster1001.eqiad.wmnet |
[production] |
| 15:09 |
<moritzm> |
installing Java 8 security updates on Hadoop systems |
[production] |
| 14:25 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubestagemaster2002.codfw.wmnet with OS bullseye |
[production] |
| 14:04 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestagemaster2002.codfw.wmnet with reason: host reimage |
[production] |
| 14:01 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on kubestagemaster2002.codfw.wmnet with reason: host reimage |
[production] |
| 13:50 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reimage for host kubestagemaster2002.codfw.wmnet with OS bullseye |
[production] |
| 13:48 |
<dcaro> |
deploy envvars-api 0.0.21 (no real changes, ci rebuild) |
[tools] |