|
2026-04-14
§
|
| 21:48 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services |
[admin] |
| 21:33 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services |
[admin] |
| 21:21 |
<andrewbogott> |
rebuilding eqiad1 rabbitmq cluster in hopes of getting some more consistent api responses |
[admin] |
| 20:01 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,nova |
[admin] |
| 19:56 |
<andrewbogott> |
restarting all nova services in eqiad1; i'm seeing inconsistent permission failures |
[admin] |
| 19:56 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova |
[admin] |
| 12:45 |
<filippo@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T419658) |
[admin] |
| 12:45 |
<filippo@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T419658) |
[admin] |
| 12:30 |
<godog> |
set maint on cloudvirt1050 - T419658 |
[admin] |
|
2026-04-13
§
|
| 19:30 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=0) |
[admin] |
| 19:06 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.ceph.roll_reboot_osds |
[admin] |
| 18:57 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) |
[admin] |
| 18:11 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.ceph.roll_reboot_osds |
[admin] |
| 18:10 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) |
[admin] |
| 18:10 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.ceph.roll_reboot_osds |
[admin] |
| 17:52 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) |
[admin] |
| 17:52 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.ceph.roll_reboot_osds |
[admin] |
| 17:52 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) |
[admin] |
| 15:17 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.ceph.roll_reboot_osds |
[admin] |
| 15:17 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) |
[admin] |
| 15:17 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.ceph.roll_reboot_osds |
[admin] |
| 15:17 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) |
[admin] |
| 15:17 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.ceph.roll_reboot_osds |
[admin] |
| 15:01 |
<andrew@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) |
[admin] |
| 15:01 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.ceph.roll_reboot_osds |
[admin] |
| 14:31 |
<godog> |
grant filippo and volans admin roles in codfw1dev |
[admin] |
|
2026-04-08
§
|
| 18:30 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services |
[admin] |
| 18:17 |
<andrewbogott> |
restarting openstack services in eqiad1 before digging into a tofu failure |
[admin] |
| 18:17 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services |
[admin] |
| 14:03 |
<godog> |
bounce nova on cloudcontrol1006 |
[admin] |
| 13:48 |
<godog> |
stop designate and memcached on all cloudcontrol1* |
[admin] |
| 13:34 |
<godog> |
leave designate processes up only on cloudcontrol1006 to ease debugging - T422646 |
[admin] |
| 11:59 |
<filippo@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for service: project,designate |
[admin] |
| 11:58 |
<filippo@cloudcumin1001> |
START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,designate |
[admin] |
| 11:54 |
<filippo@cloudcumin1001> |
END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment eqiad1 for service: project,designate |
[admin] |
| 11:53 |
<filippo@cloudcumin1001> |
START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,designate |
[admin] |
| 09:45 |
<godog> |
bounce designate on cloudcontrol |
[admin] |
| 08:14 |
<godog> |
perform more network tests on cloudrabbit1001 - T417393 |
[admin] |
| 07:57 |
<godog> |
unshut cloudcontrol1011 network interface - T417393 |
[admin] |
| 07:36 |
<godog> |
shut cloudcontrol1011 network interface - T417393 |
[admin] |
| 07:26 |
<godog> |
unshut cloudrabbit1001 network interface - T417393 |
[admin] |
| 07:00 |
<godog> |
test shutting cloudrabbit1001 network interface - T417393 |
[admin] |