1-50 of 10000 results (38ms)
2026-08-15 §
16:57 <andrewbogott> optimistically deploying a blind fix for striker issue T434684 [admin]
2026-08-13 §
16:06 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [admin]
16:05 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [admin]
16:02 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [admin]
16:01 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [admin]
16:00 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [admin]
16:00 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [admin]
12:34 <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=0) [admin]
12:23 <filippo@cloudcumin1001> START - Cookbook wmcs.ceph.roll_reboot_mons [admin]
10:00 <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=0) [admin]
09:46 <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1057' (T431682) [admin]
09:40 <filippo@cloudcumin1001> START - Cookbook wmcs.ceph.roll_reboot_osds [admin]
09:30 <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1057' (T431682) [admin]
09:18 <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1056' (T431682) [admin]
09:01 <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1056' (T431682) [admin]
09:01 <godog> put cloudvirt1064 back into aggregate network-ovs [admin]
08:42 <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1055' (T431682) [admin]
08:22 <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055' (T431682) [admin]
08:19 <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1054' (T431682) [admin]
08:02 <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1054' (T431682) [admin]
07:56 <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1051' (T431682) [admin]
07:39 <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1051' (T431682) [admin]
07:38 <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1049' (T431682) [admin]
07:29 <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1049' (T431682) [admin]
2026-08-11 §
15:20 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.roll_restart_osd_daemons (exit_code=0) (1324319) [admin]
14:58 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.roll_restart_osd_daemons (1324319) [admin]
2026-08-09 §
19:25 <andrewbogott> "ceph osd out 224" in response to "osd.224 observed stalled read indications in DB device" [admin]
2026-08-06 §
20:24 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.roll_restart_mon_daemons (exit_code=0) [admin]
20:24 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.roll_restart_mon_daemons [admin]
19:21 <andrewbogott> restarting osds one by one to pick up config changes [admin]
2026-08-05 §
19:54 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,magnum [admin]
19:53 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,magnum [admin]
02:17 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=0) (T429387) [admin]
01:37 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.roll_reboot_osds (T429387) [admin]
01:36 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) (T429387) [admin]
01:36 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.roll_reboot_osds (T429387) [admin]
2026-08-04 §
23:47 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) (T429387) [admin]
20:06 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.roll_reboot_osds (T429387) [admin]
20:03 <andrew@cloudcumin1001> END (ERROR) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=97) (T429387) [admin]
20:01 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.roll_reboot_osds (T429387) [admin]
18:59 <andrewbogott> systemctl reset-failed on cloudbackup2003. This patch should prevent future such false alarms: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1321058 [admin]
2026-08-03 §
16:52 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T431374) [admin]
16:52 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T431374) [admin]
14:50 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T431682) [admin]
14:50 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T431682) [admin]
14:50 <andrew@cloudcumin1001> END (ERROR) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=97) (T431374) [admin]
14:50 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T431374) [admin]
2026-07-28 §
13:24 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=0) (T431659) [admin]
13:14 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) [admin]
13:11 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.roll_reboot_mons (T431659) [admin]