5601-5650 of 10000 results (89ms)
2022-11-30 ยง
17:40 <hnowlan@deploy1002> helmfile [eqiad] START helmfile.d/services/api-gateway: sync [production]
17:39 <hnowlan@deploy1002> helmfile [codfw] DONE helmfile.d/services/api-gateway: sync [production]
17:39 <hnowlan@deploy1002> helmfile [codfw] START helmfile.d/services/api-gateway: sync [production]
17:31 <sukhe@cumin2002> START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS buster [production]
17:31 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2138:3312 (T323907)', diff saved to https://phabricator.wikimedia.org/P41899 and previous config saved to /var/cache/conftool/dbconfig/20221130-173103-ladsgroup.json [production]
17:29 <hnowlan@deploy1002> helmfile [staging] DONE helmfile.d/services/api-gateway: sync [production]
17:29 <hnowlan@deploy1002> helmfile [staging] START helmfile.d/services/api-gateway: sync [production]
17:05 <dancy@deploy1002> Finished scap: testing k8s deploy (duration: 07m 59s) [production]
16:57 <dancy@deploy1002> Started scap: testing k8s deploy [production]
16:55 <btullis@cumin1001> END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto analytics cluster: Roll restart of all Presto's jvm daemons. [production]
16:52 <eevans@deploy1002> helmfile [eqiad] DONE helmfile.d/services/echostore: apply [production]
16:52 <eevans@deploy1002> helmfile [eqiad] START helmfile.d/services/echostore: apply [production]
16:51 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2138:3312 (T323907)', diff saved to https://phabricator.wikimedia.org/P41898 and previous config saved to /var/cache/conftool/dbconfig/20221130-165108-ladsgroup.json [production]
16:51 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance [production]
16:50 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance [production]
16:50 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2126 (T323907)', diff saved to https://phabricator.wikimedia.org/P41897 and previous config saved to /var/cache/conftool/dbconfig/20221130-165047-ladsgroup.json [production]
16:44 <btullis@cumin1001> START - Cookbook sre.presto.roll-restart-workers for Presto analytics cluster: Roll restart of all Presto's jvm daemons. [production]
16:44 <eevans@deploy1002> helmfile [codfw] DONE helmfile.d/services/echostore: apply [production]
16:43 <eevans@deploy1002> helmfile [codfw] START helmfile.d/services/echostore: apply [production]
16:37 <dancy@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.40.0-wmf.12 refs T320517 [production]
16:36 <eevans@deploy1002> helmfile [staging] DONE helmfile.d/services/echostore: apply [production]
16:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P41896 and previous config saved to /var/cache/conftool/dbconfig/20221130-163540-ladsgroup.json [production]
16:34 <eevans@deploy1002> helmfile [staging] START helmfile.d/services/echostore: apply [production]
16:31 <btullis@cumin1001> END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto analytics cluster: Roll restart of all Presto's jvm daemons. [production]
16:25 <dancy@deploy1002> Pruned MediaWiki: 1.40.0-wmf.8 (duration: 02m 26s) [production]
16:25 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply [production]
16:25 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply [production]
16:25 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply [production]
16:24 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-int: apply [production]
16:23 <dancy@deploy1002> Finished scap: testwikis wikis to 1.40.0-wmf.12 refs T320517 (duration: 39m 53s) [production]
16:22 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
16:22 <oblivian@deploy1002> helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync [production]
16:22 <oblivian@deploy1002> helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
16:22 <oblivian@deploy1002> helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync [production]
16:22 <oblivian@deploy1002> helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync [production]
16:22 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
16:21 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply [production]
16:21 <oblivian@deploy1002> helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
16:20 <oblivian@deploy1002> helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync [production]
16:20 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply [production]
16:20 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [production]
16:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P41895 and previous config saved to /var/cache/conftool/dbconfig/20221130-162034-ladsgroup.json [production]
16:20 <btullis@cumin1001> START - Cookbook sre.presto.roll-restart-workers for Presto analytics cluster: Roll restart of all Presto's jvm daemons. [production]
16:19 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
16:19 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
16:18 <oblivian@deploy1002> helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync [production]
16:18 <oblivian@deploy1002> helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync [production]
16:17 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-int: apply [production]
16:17 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-ext: apply [production]
16:13 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-web: apply [production]