Search

On or before date

go

2021-02-28 §
04:54	<andrewbogott>	restarted redis-server on tools-redis-1003 and tools-redis-1004 in an attempt to reduce replag, no real change detected	[admin]
2021-02-27 §
00:33	<andrewbogott>	sudo cumin --timeout 500 "A:all and not O{project:clouddb-services}" 'lsb_release -c | grep -i buster && uname -r | grep -v 4.19.0-14-amd64 && reboot'	[admin]
00:28	<andrewbogott>	sudo cumin --timeout 500 "A:all and not O{project:clouddb-services}" 'lsb_release -c | grep -i buster && uname -r | grep -v 4.19.0-14-amd64 && echo reboot'	[admin]
00:09	<andrewbogott>	sudo cumin "A:all and not O{project:clouddb-services}" 'lsb_release -c | grep -i stretch && uname -r | grep -v 4.19.0-0.bpo.14-amd64 && reboot'	[admin]
2021-02-26 §
14:58	<dcaro>	[eqiad] rebooting cloudcephosd1015 (last osd \o/) for kernel upgrade (T275753)	[admin]
14:51	<dcaro>	[eqiad] rebooting cloudcephosd1014 for kernel upgrade (T275753)	[admin]
14:44	<dcaro>	[eqiad] rebooting cloudcephosd1013 for kernel upgrade (T275753)	[admin]
14:38	<dcaro>	[eqiad] rebooting cloudcephosd1012 for kernel upgrade (T275753)	[admin]
14:31	<dcaro>	[eqiad] rebooting cloudcephosd1011 for kernel upgrade (T275753)	[admin]
14:25	<dcaro>	[eqiad] rebooting cloudcephosd1010 for kernel upgrade (T275753)	[admin]
14:17	<dcaro>	[eqiad] rebooting cloudcephosd1009 for kernel upgrade (T275753)	[admin]
13:54	<dcaro>	[eqiad] downtimed alert1001 Ceph OSDs down alert until 18:00 GMT+1 as that is not under the host being rebooted (T275753)	[admin]
13:51	<dcaro>	[eqiad] rebooting cloudcephosd1008 for kernel upgrade (T275753)	[admin]
13:45	<dcaro>	[eqiad] rebooting cloudcephosd1007 for kernel upgrade (T275753)	[admin]
13:38	<dcaro>	[eqiad] rebooting cloudcephosd1006 for kernel upgrade (T275753)	[admin]
12:07	<dcaro>	[eqiad] rebooting cloudcephosd1005 for kernel upgrade (T275753)	[admin]
12:00	<arturo>	rebooting cloudcontrol1003 for kernel upgrade (T275753)	[admin]
11:42	<arturo>	rebooting cloudcontrol1004 for kernel upgrade (T275753)	[admin]
11:41	<dcaro>	[eqiad] rebooting cloudcephosd1004 for kernel upgrade (T275753)	[admin]
11:32	<dcaro>	[eqiad] rebooting cloudcephosd1003 for kernel upgrade (T275753)	[admin]
11:30	<arturo>	rebooting cloudcontrol1005 for kernel upgrade (T2	[admin]
11:26	<dcaro>	[eqiad] rebooting cloudcephosd1002 for kernel upgrade (T275753)	[admin]
11:16	<dcaro>	[eqiad] rebooting cloudcephosd1001 for kernel upgrade (T275753)	[admin]
11:11	<dcaro>	[eqiad] rebooting cloudcephmon1003 for kernel upgrade (T275753)	[admin]
11:05	<dcaro>	[eqiad] rebooting cloudcephmon1002 for kernel upgrade (T275753)	[admin]
10:59	<dcaro>	[eqiad] rebooting cloudcephmon1001 for kernel upgrade (T275753)	[admin]
10:45	<arturo>	rebooting cloudvirt1039 into a new kernel (T275753) --- spare	[admin]
10:43	<dcaro>	[codfw1dev] rebooting cloudcephmon2003-dev for kernel upgrade (T275753)	[admin]
10:38	<dcaro>	[codfw1dev] rebooting cloudcephmon2002-dev for kernel upgrade (T275753)	[admin]
10:29	<dcaro>	[codfw1dev] rebooting cloudcephmon2001-dev for kernel upgrade (T275753)	[admin]
10:24	<arturo>	[codfw1dev] purge old kernel packages on cloudvirt2003-dev to force boot into a new kernel (T275753)	[admin]
10:11	<arturo>	[codfw1dev] manually creating /boot/grub/ on cloudvirt2003-dev to allow update-grub2 to run (so it can reboot into a new kernel) (T275753)	[admin]
10:11	<dcaro>	[codfw1dev] rebooting cloudcephosd2003-dev for kernel upgrade (T275753)	[admin]
10:05	<dcaro>	[codfw1dev] rebooting cloudcephosd2002-dev for kernel upgrade (T275753)	[admin]
10:01	<arturo>	[codfw1dev] rebooting cloudvirt200X-dev for kernel upgrade (T275753)	[admin]
09:59	<arturo>	[codfw1dev] rebooting cloudweb2001-dev for kernel upgrade (T275753)	[admin]
09:53	<arturo>	[codfw1dev] rebooting cloudservices2003-dev for kernel upgrade (T275753)	[admin]
09:51	<arturo>	[codfw1dev] rebooting cloudservices2002-dev for kernel upgrade (T275753)	[admin]
09:45	<arturo>	[codfw1dev] rebooting cloudcontrol2004-dev for kernel upgrade (T275753)	[admin]
09:44	<arturo>	[codfw1dev] rebooting cloudbackup[2001-2002].codfw.wmnet for kernel upgrade (T275753)	[admin]
09:43	<dcaro>	[codfw1dev] rebooting cloudcephosd2001-dev for kernel upgrade (T275753)	[admin]
09:41	<arturo>	[codfw1dev] rebooting cloudcontrol2003-dev for kernel upgrade (T275753)	[admin]
09:33	<arturo>	[codfw1dev] rebooting cloudcontrol2001-dev for kernel upgrade (T275753)	[admin]
2021-02-25 §
14:56	<arturo>	deployed wmcs-netns-events daemon to all cloudnet servers (T275483)	[admin]
2021-02-24 §
11:07	<arturo>	force-reboot cloudmetrics1002, add icinga downtime for 2 hours. Investigating some server issue	[admin]
00:17	<bstorm>	set --property hw_scsi_model=virtio-scsi and --property hw_disk_bus=scsi on the main stretch image in glance on eqiad1 T275430	[admin]
2021-02-23 §
22:43	<bstorm>	set --property hw_scsi_model=virtio-scsi and --property hw_disk_bus=scsi on the main buster image in glance on eqiad1 T275430	[admin]
20:36	<andrewbogott>	adding r/o access to the eqiad1-glance-images ceph pool for the client.eqiad1-compute for T275430	[admin]
10:49	<arturo>	rebooting clounet1004 into new kernel from buster-bpo (T271058)	[admin]
10:48	<arturo>	installing linux-image-amd64 from buster-bpo 5.10.13-1~bpo10+1 in cloudnet1004 (T271058)	[admin]