|
2024-06-25
§
|
| 03:55 |
<mwpresync@deploy1002> |
Finished scap: testwikis wikis to 1.43.0-wmf.11 refs T366956 (duration: 52m 19s) |
[production] |
| 03:42 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) |
[admin] |
| 03:39 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.restart_openstack |
[admin] |
| 03:17 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) |
[admin] |
| 03:12 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.restart_openstack |
[admin] |
| 03:12 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) |
[admin] |
| 03:06 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.restart_openstack |
[admin] |
| 03:03 |
<mwpresync@deploy1002> |
Started scap: testwikis wikis to 1.43.0-wmf.11 refs T366956 |
[production] |
| 01:48 |
<brett> |
Running authdns-update on dns1004 to pool eqsin - T365763 |
[production] |
| 01:43 |
<brett@puppetmaster1001> |
conftool action : set/pooled=yes; selector: cluster=cache_text,dc=eqsin |
[production] |
| 01:40 |
<brett> |
Removing downtime for cp[5017-5024] as nvme drives are installed and hosts back online - T365763 |
[production] |
| 00:43 |
<sukhe> |
[correction of command] sudo pkill ffmpeg: mw1438, high CPU usage, ffmpeg processes |
[production] |
| 00:43 |
<sukhe> |
sudo pkill mpeg: mw1438, high CPU usage, ffmpeg processes |
[production] |
| 00:01 |
<brett@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 8 hosts with reason: T365763 |
[production] |
| 00:01 |
<brett@cumin2002> |
START - Cookbook sre.hosts.downtime for 4:00:00 on 8 hosts with reason: T365763 |
[production] |
|
2024-06-24
§
|
| 23:02 |
<brett> |
Running authdns-update on dns1004 to depool eqsin - T365763 |
[production] |
| 23:01 |
<bd808> |
Removed matanya's "reader" right per T368330 |
[releng] |
| 23:01 |
<bd808> |
Removed matanya's "reader" right per T368330 |
[deployment-prep] |
| 23:00 |
<cwhite@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts logstash2003.codfw.wmnet |
[production] |
| 23:00 |
<cwhite@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 23:00 |
<cwhite@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002" |
[production] |
| 22:59 |
<bd808> |
Removed matanya's "member" right per T368330 |
[video] |
| 22:57 |
<cwhite@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002" |
[production] |
| 22:53 |
<cwhite@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:46 |
<cwhite@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts logstash2003.codfw.wmnet |
[production] |
| 22:46 |
<cwhite@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts logstash2002.codfw.wmnet |
[production] |
| 22:46 |
<cwhite@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:46 |
<cwhite@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002" |
[production] |
| 22:41 |
<cwhite@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002" |
[production] |
| 22:38 |
<cwhite@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:26 |
<cwhite@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts logstash2002.codfw.wmnet |
[production] |
| 22:26 |
<cwhite@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts logstash2001.codfw.wmnet |
[production] |
| 22:26 |
<cwhite@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 22:26 |
<cwhite@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002" |
[production] |
| 22:24 |
<cwhite@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: logstash2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - cwhite@cumin2002" |
[production] |
| 22:18 |
<cwhite@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 22:11 |
<cwhite@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts logstash2001.codfw.wmnet |
[production] |
| 21:34 |
<inflatador> |
bking@alerts1001 uninstall deb pkg `ripgrep` T368107 |
[production] |
| 21:03 |
<btullis@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-redacteddb1001.eqiad.wmnet |
[production] |
| 21:00 |
<eevans@cumin1002> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-codfw: Apply Cassandra upgrade to 4.1.5 — T354970 - eevans@cumin1002 |
[production] |
| 20:56 |
<andrewbogott> |
rebooting tools-k8s-worker-nfs-36; it has lots of stuck processes which somehow didn't get unstuck when we did the post-nfs-migration reboots. |
[tools] |
| 20:53 |
<btullis@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host an-redacteddb1001.eqiad.wmnet |
[production] |
| 20:36 |
<inflatador> |
bking@alert1001 install `ripgrep` deb pkg T368107 |
[production] |
| 20:22 |
<ladsgroup@deploy1002> |
Synchronized php-1.43.0-wmf.10/includes/libs/rdbms/loadbalancer/LoadBalancer.php: (no justification provided) (duration: 11m 04s) |
[production] |
| 20:21 |
<mutante> |
snapsho1017 - systemctl mask commonsrdf-dump ; systemctl mask commonsjson-dump T368098 |
[production] |
| 20:18 |
<taavi> |
taavi@snapshot1017 ~ $ sudo systemctl stop commons*.service |
[production] |
| 20:09 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1) |
[metricsinfra] |
| 20:01 |
<andrew@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1056.eqiad.wmnet with OS bookworm |
[production] |
| 20:01 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate False, for hosts list: ['cloudvirt1056'] |
[cloudvirt-canary] |
| 20:00 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1056'] |
[cloudvirt-canary] |