2551-2600 of 10000 results (52ms)
2022-03-10 ยง
19:46 <volans@cumin2002> START - Cookbook sre.misc-clusters.sretest rolling restart_daemons on A:sretest [production]
19:44 <dancy@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.38.0-wmf.25 refs T300201 [production]
19:44 <volans> uploaded spicerack_2.3.2 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia [production]
19:33 <dduvall@deploy1002> helmfile [eqiad] DONE helmfile.d/services/blubberoid: apply [production]
19:32 <dduvall@deploy1002> helmfile [eqiad] START helmfile.d/services/blubberoid: apply [production]
19:32 <dduvall@deploy1002> helmfile [codfw] DONE helmfile.d/services/blubberoid: apply [production]
19:31 <dduvall@deploy1002> helmfile [codfw] START helmfile.d/services/blubberoid: apply [production]
19:29 <dduvall@deploy1002> helmfile [staging] DONE helmfile.d/services/blubberoid: apply [production]
19:29 <dduvall@deploy1002> helmfile [staging] START helmfile.d/services/blubberoid: apply [production]
19:21 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
19:21 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
19:21 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
19:16 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
19:11 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
19:10 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
19:10 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
19:09 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
19:07 <jayme@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventgate-main: apply [production]
19:06 <jayme@deploy1002> helmfile [eqiad] START helmfile.d/services/eventgate-main: apply [production]
19:06 <jayme@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventgate-logging-external: apply [production]
19:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1167 (T300775)', diff saved to https://phabricator.wikimedia.org/P22352 and previous config saved to /var/cache/conftool/dbconfig/20220310-190544-marostegui.json [production]
19:05 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
19:05 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
19:05 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1167.eqiad.wmnet with reason: Maintenance [production]
19:05 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1167.eqiad.wmnet with reason: Maintenance [production]
19:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3318 (T300775)', diff saved to https://phabricator.wikimedia.org/P22351 and previous config saved to /var/cache/conftool/dbconfig/20220310-190530-marostegui.json [production]
19:04 <jayme@deploy1002> helmfile [eqiad] START helmfile.d/services/eventgate-logging-external: apply [production]
19:04 <jayme@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply [production]
19:02 <jayme@deploy1002> helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply [production]
19:02 <jayme@deploy1002> helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply [production]
19:01 <jayme@deploy1002> helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply [production]
19:00 <jayme@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-main: apply [production]
18:59 <jayme@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-main: apply [production]
18:59 <jayme@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: apply [production]
18:58 <jayme@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-logging-external: apply [production]
18:58 <jayme@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply [production]
18:57 <jayme@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply [production]
18:57 <jayme@deploy1002> helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply [production]
18:56 <jayme@deploy1002> helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply [production]
18:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3318', diff saved to https://phabricator.wikimedia.org/P22350 and previous config saved to /var/cache/conftool/dbconfig/20220310-185025-marostegui.json [production]
18:46 <jayme@deploy1002> helmfile [staging] DONE helmfile.d/services/eventgate-main: apply [production]
18:43 <jayme@deploy1002> helmfile [staging] START helmfile.d/services/eventgate-main: apply [production]
18:43 <jayme@deploy1002> helmfile [staging] DONE helmfile.d/services/eventgate-logging-external: apply [production]
18:41 <jayme@deploy1002> helmfile [staging] START helmfile.d/services/eventgate-logging-external: apply [production]
18:41 <jayme@deploy1002> helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: apply [production]
18:40 <moritzm> restarting thumbor to pick up tiff security updates [production]
18:40 <jayme@deploy1002> helmfile [staging] START helmfile.d/services/eventgate-analytics-external: apply [production]
18:40 <jayme@deploy1002> helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply [production]
18:39 <jayme@deploy1002> helmfile [staging] START helmfile.d/services/eventgate-analytics: apply [production]
18:36 <moritzm> installing tiff security updates [production]