6851-6900 of 10000 results (107ms)
2024-06-24 §
17:45 <swfrench@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-int: apply [production]
17:44 <swfrench@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply [production]
17:44 <andrew@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1055.eqiad.wmnet with reason: host reimage [production]
17:43 <swfrench@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-int: apply [production]
17:34 <sukhe@puppetmaster1001> conftool action : set/pooled=no; selector: cluster=apus,dc=codfw [production]
17:33 <swfrench@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply [production]
17:32 <swfrench@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-int: apply [production]
17:28 <andrew@cumin1002> START - Cookbook sre.hosts.reimage for host cloudvirt1055.eqiad.wmnet with OS bookworm [production]
17:28 <swfrench@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply [production]
17:27 <swfrench@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-int: apply [production]
17:23 <sukhe> restart pybal on lvs2013 [production]
17:20 <swfrench@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
17:19 <swfrench@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
17:18 <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching P{P:cassandra%rack = "b"} and A:restbase and A:eqiad: Apply Cassandra upgrade to 4.1.5 — T354970 - eevans@cumin1002 [production]
17:13 <andrew@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1054.eqiad.wmnet with OS bookworm [production]
17:13 <sukhe> restart pybal on lvs1020 and lvs1019 [production]
17:09 <swfrench@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
17:08 <swfrench@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
16:55 <cdanis@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply [production]
16:51 <cdanis@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply [production]
16:49 <cdanis@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply [production]
16:48 <sukhe> restart pybal on lvs1020 [production]
16:47 <cdanis@deploy1002> helmfile [codfw] START helmfile.d/services/mw-api-ext: apply [production]
16:44 <andrew@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1054.eqiad.wmnet with reason: host reimage [production]
16:41 <andrew@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1054.eqiad.wmnet with reason: host reimage [production]
16:33 <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase[1031,1034-1036].eqiad.wmnet: Apply Cassandra upgrade to 4.1.5 — T354970 - eevans@cumin1002 [production]
16:27 <andrew@cumin1002> START - Cookbook sre.hosts.reimage for host cloudvirt1054.eqiad.wmnet with OS bookworm [production]
16:20 <sukhe> restart pybal on lvs1020 [production]
16:01 <dancy@deploy1002> Installation of scap version "4.89.0" completed for 1 hosts [production]
16:00 <dancy@deploy1002> Installing scap version "4.89.0" for 1 hosts [production]
15:59 <sukhe> restart pybal on lvs1020 [production]
15:59 <dancy@deploy1002> Installing scap version "4.89.0" for 248 hosts [production]
15:57 <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase[1031,1034-1036].eqiad.wmnet: Apply Cassandra upgrade to 4.1.5 — T354970 - eevans@cumin1002 [production]
15:50 <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs1010.eqiad.wmnet: Apply Cassandra upgrade to 4.1.5 — T354970 - eevans@cumin1002 [production]
15:49 <sukhe> restart pybal on lvs1020 [production]
15:43 <vgutierrez> updated termination_state cache haproxy metrics, expect higher CD and CR rates - T367963 [production]
15:42 <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching aqs1010.eqiad.wmnet: Apply Cassandra upgrade to 4.1.5 — T354970 - eevans@cumin1002 [production]
15:29 <elukey@deploy1002> helmfile [eqiad] DONE helmfile.d/services/wikifeeds: sync [production]
15:29 <elukey@deploy1002> helmfile [eqiad] START helmfile.d/services/wikifeeds: sync [production]
15:20 <elukey@deploy1002> helmfile [codfw] DONE helmfile.d/services/wikifeeds: sync [production]
15:20 <elukey@deploy1002> helmfile [codfw] START helmfile.d/services/wikifeeds: sync [production]
15:17 <cgoubert@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply [production]
15:16 <elukey@deploy1002> helmfile [staging] DONE helmfile.d/services/wikifeeds: sync [production]
15:16 <elukey@deploy1002> helmfile [staging] START helmfile.d/services/wikifeeds: sync [production]
15:15 <cgoubert@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply [production]
15:15 <cgoubert@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply [production]
15:13 <cgoubert@deploy1002> helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply [production]
15:11 <mvernon@cumin1002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-secondary-eqiad and A:lvs (T279621) [production]
15:11 <claime> Enabling statsd-exporter on mw-jobrunner - T365265 [production]
15:11 <mvernon@cumin1002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary-eqiad and A:lvs (T279621) [production]