251-300 of 10000 results (87ms)
2024-06-25 §
01:48 <brett> Running authdns-update on dns1004 to pool eqsin - T365763 [production]
01:43 <brett@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=cache_text,dc=eqsin [production]
01:40 <brett> Removing downtime for cp[5017-5024] as nvme drives are installed and hosts back online - T365763 [production]
00:43 <sukhe> [correction of command] sudo pkill ffmpeg: mw1438, high CPU usage, ffmpeg processes [production]
00:43 <sukhe> sudo pkill mpeg: mw1438, high CPU usage, ffmpeg processes [production]
00:01 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 8 hosts with reason: T365763 [production]
00:01 <brett@cumin2002> START - Cookbook sre.hosts.downtime for 4:00:00 on 8 hosts with reason: T365763 [production]
2024-06-24 §
23:02 <brett> Running authdns-update on dns1004 to depool eqsin - T365763 [production]
23:00 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts logstash2003.codfw.wmnet [production]
23:00 <cwhite@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
23:00 <cwhite@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002" [production]
22:57 <cwhite@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002" [production]
22:53 <cwhite@cumin2002> START - Cookbook sre.dns.netbox [production]
22:46 <cwhite@cumin2002> START - Cookbook sre.hosts.decommission for hosts logstash2003.codfw.wmnet [production]
22:46 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts logstash2002.codfw.wmnet [production]
22:46 <cwhite@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:46 <cwhite@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002" [production]
22:41 <cwhite@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002" [production]
22:38 <cwhite@cumin2002> START - Cookbook sre.dns.netbox [production]
22:26 <cwhite@cumin2002> START - Cookbook sre.hosts.decommission for hosts logstash2002.codfw.wmnet [production]
22:26 <cwhite@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts logstash2001.codfw.wmnet [production]
22:26 <cwhite@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:26 <cwhite@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002" [production]
22:24 <cwhite@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002" [production]
22:18 <cwhite@cumin2002> START - Cookbook sre.dns.netbox [production]
22:11 <cwhite@cumin2002> START - Cookbook sre.hosts.decommission for hosts logstash2001.codfw.wmnet [production]
21:34 <inflatador> bking@alerts1001 uninstall deb pkg `ripgrep` T368107 [production]
21:03 <btullis@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-redacteddb1001.eqiad.wmnet [production]
21:00 <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-codfw: Apply Cassandra upgrade to 4.1.5 — T354970 - eevans@cumin1002 [production]
20:53 <btullis@cumin1002> START - Cookbook sre.hosts.reboot-single for host an-redacteddb1001.eqiad.wmnet [production]
20:36 <inflatador> bking@alert1001 install `ripgrep` deb pkg T368107 [production]
20:22 <ladsgroup@deploy1002> Synchronized php-1.43.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php: (no justification provided) (duration: 11m 04s) [production]
20:21 <mutante> snapsho1017 - systemctl mask commonsrdf-dump ; systemctl mask commonsjson-dump T368098 [production]
20:18 <taavi> taavi@snapshot1017 ~ $ sudo systemctl stop commons*.service [production]
20:01 <andrew@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1056.eqiad.wmnet with OS bookworm [production]
19:35 <andrew@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1056.eqiad.wmnet with reason: host reimage [production]
19:32 <andrew@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1056.eqiad.wmnet with reason: host reimage [production]
19:27 <bking@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow: apply [production]
19:27 <bking@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow: apply [production]
19:26 <bking@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow: apply [production]
19:26 <bking@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow: apply [production]
19:08 <mutante> LDAP - added daphnesmit to group 'wmf' - Phabricator: added dsmit-wmf to WMF-NDA group T368140 [production]
19:02 <sukhe> ms-fe1009: restart swift-proxy: T360913 [production]
18:59 <mutante> ms-fe1011 - restarted swift-proxy [production]
18:53 <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-codfw: Apply Cassandra upgrade to 4.1.5 — T354970 - eevans@cumin1002 [production]
18:52 <eevans@cumin1002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 15 hosts [production]
18:52 <eevans@cumin1002> START - Cookbook sre.hosts.remove-downtime for 15 hosts [production]
18:50 <eevans@cumin1002> END (ERROR) - Cookbook sre.cassandra.roll-restart (exit_code=97) for nodes matching A:restbase-eqiad: Apply Cassandra upgrade to 4.1.5 — T354970 - eevans@cumin1002 [production]
18:50 <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Apply Cassandra upgrade to 4.1.5 — T354970 - eevans@cumin1002 [production]
18:50 <sukhe> sudo cumin -s1 -b60 'ms-fe1010*,ms-fe1013*' 'systemctl restart swift-proxy' [production]