3451-3500 of 4096 results (24ms)
2020-12-18 §
16:54 <dcaro> finished cleaning up the dangling snapshots from cloudvirt1023 (T270478) [admin]
16:51 <dcaro> removing dangling rbd snapshots (for backups on cloudvirt1023) (T270478) [admin]
16:47 <dcaro> finished cleaning up the dangling snapshots from cloudvirt1024, freed ~12% of the capacity (T270478) [admin]
16:21 <dcaro> removing dangling rbd snapshots (for backups on cloudvirt1024) (T270478) [admin]
16:13 <andrewbogott> setting autoscale to 'off' for both ceph pools (eqiad1-compute and eqiad1-glance-images) because we like how things are set and the autoscaler does not [admin]
10:33 <dcaro> purging rbd snapshots for image fc6fb78b-4515-4dcc-8254-591b9fe01762 (T270478) [admin]
2020-12-17 §
22:17 <andrewbogott> correction to above, set the pg and pgp to 1024 for eqiad1-glance-images [admin]
22:16 <andrewbogott> setting pgp number to 8192 for eqiad1-compute (a 4x increase) and 2048 for eqiad1-glance-images (also a 4x increase) T270305 (same as pg) [admin]
22:14 <andrewbogott> setting pg number to 8192 for eqiad1-compute (a 4x increase) and 2048 for eqiad1-glance-images (also a 4x increase) T270305 [admin]
22:10 <andrewbogott> setting autoscale to 'warn' for both ceph pools (eqiad1-compute and eqiad1-glance-images) [admin]
2020-12-16 §
09:31 <dcaro> removing invalid backups from cloudvirt1024 (196 in total) (T269419) [admin]
2020-12-14 §
17:41 <dcaro> The removal freed ~12GB (still 100% usage :S) (T269419) [admin]
17:36 <dcaro> removing invalid backups that have a valid copy (T269419) [admin]
15:43 <dcaro> Merging the tagging for vm backups (T267195) [admin]
09:45 <arturo> icinga downtime cloudvirt1024 for 6 days (T269419) [admin]
2020-12-13 §
09:11 <_dcaro> running backup purge script on cloudvirt1024 (T269419) [admin]
2020-12-10 §
23:36 <bstorm> cleaned up the logs for haproxy on cloudcontrol1003 by deleting all the gzipped ones and truncating the .1 file [admin]
11:56 <dcaro> Freed some space on cloudvirt1024 by running the purge script (T269419) [admin]
09:17 <dcaro> removing leaked dns record discordwiki.eqiad.wmflabs (clinic duty) [admin]
2020-12-08 §
18:01 <dcaro> Host cloudvirt1030 up and running (T216195) [admin]
15:59 <dcaro> Re-imaging host cloudvirt1030 (T216195) [admin]
14:18 <dcaro> Host online cloudvirt1029 (T216195) [admin]
14:13 <dcaro> Host re-imaged, doing tests cloudvirt1029 (T216195) [admin]
12:14 <dcaro> Re-imaging cloudvirt1029 (T216195) [admin]
2020-12-07 §
18:33 <andrewbogott> putting cloudvirt1023 back into service T269467 [admin]
15:55 <andrewbogott> reimaging cloudvirt1028 for T216195 [admin]
14:49 <dcaro> Re-imaging cloudvirt1027 (T216195) [admin]
2020-12-05 §
00:35 <andrewbogott> moving cloudvirt1023 back into maintenance because T269467 continues to puzzle [admin]
2020-12-04 §
22:33 <andrewbogott> moving cloudvirt1023 back into the ceph aggregate; it doesn't need upgrades after all T269467 [admin]
22:23 <andrewbogott> moving cloudvirt1023 out of the ceph aggregate and into maintenance for T269467 [admin]
21:06 <andrewbogott> putting cloudvirt1025 and 1026 back into service because I'm pretty sure they're fixed. T269313 [admin]
12:12 <arturo> manually running `wmcs-purge-backups` again on cloudvirt1024 (T269419) [admin]
11:25 <arturo> icinga downtime cloudvirt1024 for 6 days, to avoid paging noises (T269419) [admin]
11:25 <arturo> last log line referencing cloudvirt1024 is a mistake (T269313) [admin]
11:24 <arturo> icinga downtime cloudvirt1024 for 6 days, to avoid paging noises (T269313) [admin]
10:28 <arturo> manually running `wmcs-purge-backups` on cloudvirt1024 (T269419) [admin]
10:23 <arturo> setting expiration to 2020-12-03 to the oldest backy snapshot of every VM in cloudvirt1024 (T269419) [admin]
09:54 <arturo> icinga downtime cloudvirt1025 for 6 days (T269313) [admin]
2020-12-03 §
23:21 <andrewbogott> removing all osds on cloudcephosd1004 for rebuild, T268746 [admin]
21:45 <andrewbogott> removing all osds on cloudcephosd1005 for rebuild, T268746 [admin]
19:51 <andrewbogott> removing all osds on cloudcephosd1006 for rebuild, T268746 [admin]
17:01 <arturo> icinga downtime cloudvirt1025 for 48h to debug network issue T269313 [admin]
16:56 <arturo> rebooting cloudvirt1025 to debug network issue T269313 [admin]
16:38 <dcaro> Rimaging cloudvirt1026 (T216195) [admin]
13:24 <andrewbogott> removing all osds on cloudcephosd1008 for rebuild, T268746 [admin]
02:55 <andrewbogott> removing all osds on cloudcephosd1009 for rebuild, T268746 [admin]
2020-12-02 §
20:03 <andrewbogott> removing all osds on cloudcephosd1010 for rebuild, T268746 [admin]
17:25 <arturo> [15:51] failovering neutron virtual router in eqiad1 (T268335) [admin]
15:36 <arturo> conntrackd is now up and running in cloudnet1003/1004 nodes (T268335) [admin]
15:33 <arturo> [codfw1dev] conntrackd is now up and running in cloudnet200x-dev nodes (T268335) [admin]