4401-4450 of 10000 results (96ms)
2022-11-30 ยง
16:50 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance [production]
16:50 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2126 (T323907)', diff saved to https://phabricator.wikimedia.org/P41897 and previous config saved to /var/cache/conftool/dbconfig/20221130-165047-ladsgroup.json [production]
16:44 <btullis@cumin1001> START - Cookbook sre.presto.roll-restart-workers for Presto analytics cluster: Roll restart of all Presto's jvm daemons. [production]
16:44 <eevans@deploy1002> helmfile [codfw] DONE helmfile.d/services/echostore: apply [production]
16:43 <eevans@deploy1002> helmfile [codfw] START helmfile.d/services/echostore: apply [production]
16:37 <dancy@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.40.0-wmf.12 refs T320517 [production]
16:36 <eevans@deploy1002> helmfile [staging] DONE helmfile.d/services/echostore: apply [production]
16:35 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P41896 and previous config saved to /var/cache/conftool/dbconfig/20221130-163540-ladsgroup.json [production]
16:34 <eevans@deploy1002> helmfile [staging] START helmfile.d/services/echostore: apply [production]
16:31 <btullis@cumin1001> END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto analytics cluster: Roll restart of all Presto's jvm daemons. [production]
16:25 <dancy@deploy1002> Pruned MediaWiki: 1.40.0-wmf.8 (duration: 02m 26s) [production]
16:25 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply [production]
16:25 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply [production]
16:25 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply [production]
16:24 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-int: apply [production]
16:23 <dancy@deploy1002> Finished scap: testwikis wikis to 1.40.0-wmf.12 refs T320517 (duration: 39m 53s) [production]
16:22 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
16:22 <oblivian@deploy1002> helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync [production]
16:22 <oblivian@deploy1002> helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
16:22 <oblivian@deploy1002> helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync [production]
16:22 <oblivian@deploy1002> helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync [production]
16:22 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
16:21 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply [production]
16:21 <oblivian@deploy1002> helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync [production]
16:20 <oblivian@deploy1002> helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync [production]
16:20 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply [production]
16:20 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [production]
16:20 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P41895 and previous config saved to /var/cache/conftool/dbconfig/20221130-162034-ladsgroup.json [production]
16:20 <btullis@cumin1001> START - Cookbook sre.presto.roll-restart-workers for Presto analytics cluster: Roll restart of all Presto's jvm daemons. [production]
16:19 <oblivian@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
16:19 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
16:18 <oblivian@deploy1002> helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync [production]
16:18 <oblivian@deploy1002> helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync [production]
16:17 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-int: apply [production]
16:17 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-ext: apply [production]
16:13 <oblivian@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-web: apply [production]
16:09 <oblivian@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-web: apply [production]
16:07 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
16:05 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2126 (T323907)', diff saved to https://phabricator.wikimedia.org/P41892 and previous config saved to /var/cache/conftool/dbconfig/20221130-160527-ladsgroup.json [production]
16:03 <oblivian@deploy1002> helmfile [codfw] START helmfile.d/services/mw-web: apply [production]
15:57 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
15:54 <akosiaris@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on thumbor2004.codfw.wmnet with reason: work on iDrac [production]
15:54 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on ores2009.codfw.wmnet with reason: DCOps maintenance [production]
15:54 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:30:00 on ores2009.codfw.wmnet with reason: DCOps maintenance [production]
15:54 <akosiaris@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on thumbor2004.codfw.wmnet with reason: work on iDrac [production]
15:52 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
15:51 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
15:51 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
15:49 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
15:49 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2126 (T323907)', diff saved to https://phabricator.wikimedia.org/P41891 and previous config saved to /var/cache/conftool/dbconfig/20221130-154917-ladsgroup.json [production]