|
2023-07-04
§
|
| 10:52 |
<jayme@deploy1002> |
helmfile [staging] OK helmfile.d/services/recommendation-api: -i apply |
[production] |
| 10:52 |
<jayme@deploy1002> |
helmfile [staging] OK helmfile.d/services/machinetranslation: -i apply |
[production] |
| 10:52 |
<jayme@deploy1002> |
helmfile [staging] OK helmfile.d/services/sessionstore: -i apply |
[production] |
| 10:52 |
<jayme@deploy1002> |
helmfile [staging] OK helmfile.d/services/developer-portal: -i apply |
[production] |
| 10:52 |
<jayme@deploy1002> |
helmfile [staging] OK helmfile.d/services/eventgate-logging-external: -i apply |
[production] |
| 10:52 |
<jayme@deploy1002> |
helmfile [staging] OK helmfile.d/services/rest-gateway: -i apply |
[production] |
| 10:31 |
<btullis> |
beginning hdfs datanode rolling restart with `sudo cumin -b 2 -p 80 -s 120 A:hadoop-worker 'systemctl restart hadoop-hdfs-datanode'` |
[analytics] |
| 10:20 |
<jayme@deploy1002> |
helmfile [staging] FAIL (3) helmfile.d/services/mw-api-int: -i apply |
[production] |
| 10:20 |
<jayme@deploy1002> |
helmfile [staging] OK helmfile.d/services/rest-gateway: -i apply |
[production] |
| 10:10 |
<btullis> |
btullis@an-master1001:~$ sudo systemctl start hadoop-hdfs-namenode |
[analytics] |
| 10:05 |
<jayme@deploy1002> |
helmfile [staging] START helmfile.d/services/eventgate-logging-external: apply |
[production] |
| 10:05 |
<jayme@deploy1002> |
helmfile [staging] START helmfile.d/services/datahub: apply on main |
[production] |
| 10:05 |
<jayme@deploy1002> |
helmfile [staging] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 10:05 |
<jayme@deploy1002> |
helmfile [staging] START helmfile.d/services/rest-gateway: apply |
[production] |
| 10:04 |
<jnuche@deploy1002> |
rebuilt and synchronized wikiversions files: group0 wikis to 1.41.0-wmf.16 refs T340244 |
[production] |
| 10:00 |
<btullis> |
roll-restarting journal nodes with 30 seconds between each one: `sudo cumin -b 1 -p 100 -s 30 A:hadoop-hdfs-journal 'systemctl restart hadoop-hdfs-journalnode'` |
[analytics] |
| 09:55 |
<jnuche@deploy1002> |
Pruned MediaWiki: 1.41.0-wmf.13 (duration: 02m 11s) |
[production] |
| 09:52 |
<jnuche@deploy1002> |
Finished scap: testwikis wikis to 1.41.0-wmf.16 refs T340244 (duration: 50m 51s) |
[production] |
| 09:48 |
<cgoubert@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop: apply |
[production] |
| 09:47 |
<cgoubert@deploy1002> |
helmfile [eqiad] START helmfile.d/services/changeprop: apply |
[production] |
| 09:47 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/changeprop: apply |
[production] |
| 09:46 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/changeprop: apply |
[production] |
| 09:46 |
<cgoubert@deploy1002> |
helmfile [staging] DONE helmfile.d/services/changeprop: apply |
[production] |
| 09:46 |
<cgoubert@deploy1002> |
helmfile [staging] START helmfile.d/services/changeprop: apply |
[production] |
| 09:43 |
<cgoubert@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:42 |
<cgoubert@deploy1002> |
helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:42 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:42 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:42 |
<cgoubert@deploy1002> |
helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:41 |
<cgoubert@deploy1002> |
helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:38 |
<jayme> |
updated envoyproxy to 1.23.10 on all nodes - T300324 |
[production] |
| 09:37 |
<cgoubert@deploy1002> |
helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:37 |
<cgoubert@deploy1002> |
helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:36 |
<cgoubert@deploy1002> |
helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:36 |
<cgoubert@deploy1002> |
helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply |
[production] |
| 09:29 |
<btullis> |
restarting the yarn restart with `sudo cumin -b 5 -p 80 -s 30 A:hadoop-worker 'systemctl restart hadoop-yarn-nodemanager'` |
[analytics] |
| 09:07 |
<btullis@cumin1001> |
END (FAIL) - Cookbook sre.hadoop.roll-restart-workers (exit_code=99) restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade. |
[production] |
| 09:02 |
<jnuche@deploy1002> |
Started scap: testwikis wikis to 1.41.0-wmf.16 refs T340244 |
[production] |
| 08:57 |
<btullis> |
executing `cookbook sre.hadoop.roll-restart-workers analytics` |
[analytics] |
| 08:56 |
<btullis@cumin1001> |
START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade. |
[production] |
| 08:09 |
<wm-bot> |
<lucaswerkmeister> Double IRC messages to other bridges |
[tools.bridgebot] |
| 06:31 |
<wm-bot> |
<lokal-profil> Source code is now available at https://gist.github.com/lokal-profil/2e59b4b07d449a2c9a8ed98429787d80 |
[tools.slumpartikel] |
| 06:16 |
<wm-bot> |
<lokal-profil> Adding creator and license to source file and moving username out of source |
[tools.slumpartikel] |