1-50 of 669 results (19ms)
2020-12-28 §
12:32 <arturo> stop doing backups for the dumps project https://gerrit.wikimedia.org/r/c/operations/puppet/+/652182 (T260692) [admin]
12:32 <arturo> stop doing backups for the dumps project https://gerrit.wikimedia.org/r/c/operations/puppet/+/652182 (T260682) [admin]
12:23 <arturo> icinga downtime cloudvirt1026 disk space check until january 5 (T260692) [admin]
06:15 <andrewbogott> restarting designate-central on cloudservices1003/1004. I'm pretty sure they're distressed because of DB lag but it's worth a try [admin]
2020-12-23 §
15:38 <andrewbogott> restarting rabbitmq on cloudcontrol1004; suspected leaks [admin]
15:33 <andrewbogott> restarting each cloudcontrol galera node in turn to see if that quiets down the syncing warnings [admin]
12:08 <arturo> move memory out of the swap in cloudcontrol1004 by disabling/enabling it (1Gb swap was being used) [admin]
2020-12-22 §
15:30 <dcaro> cleaning up 6778 dangling snapshots for glance images in eqiad (T270478) [admin]
13:51 <dcaro> merged patch to move wikidumpparse backups to cloudvirt1025 to free space on cloudvirt1026 [admin]
2020-12-19 §
16:18 <dcaro> gzipped a bunch of logs on cloudvirt1004 due to / being out of space [admin]
00:14 <bstorm> truncated /var/log/debug.1 on cloudcontrol1003 which appears to be the exact same content as the user.log files anyway [admin]
00:10 <bstorm> truncated /var/log/daemon.log.1 and the haproxy log [admin]
00:02 <bstorm> truncated /var/log/messages.1 on cloudcontrol1003 [admin]
2020-12-18 §
23:53 <bstorm> truncated haproxy.log.1 on cloudcontrol1003 [admin]
20:46 <andrewbogott> setting pg and pgp number to 4096 for eqiad1-compute as joachim thinks 8192 might be too much T270305 [admin]
17:09 <dcaro> finished cleaning up the dangling snapshots from cloudvirt1026 (T270478) [admin]
17:08 <dcaro> removing dangling rbd snapshots (for backups on cloudvirt1026) (T270478) [admin]
17:06 <dcaro> finished cleaning up the dangling snapshots from cloudvirt1025 (T270478) [admin]
17:05 <dcaro> removing dangling rbd snapshots (for backups on cloudvirt1025) (T270478) [admin]
17:00 <dcaro> finished cleaning up the dangling snapshots from cloudvirt1021 (T270478) [admin]
16:58 <dcaro> removing dangling rbd snapshots (for backups on cloudvirt1021) (T270478) [admin]
16:56 <dcaro> finished cleaning up the dangling snapshots from cloudvirt1022 (T270478) [admin]
16:55 <dcaro> removing dangling rbd snapshots (for backups on cloudvirt1022) (T270478) [admin]
16:54 <dcaro> finished cleaning up the dangling snapshots from cloudvirt1023 (T270478) [admin]
16:51 <dcaro> removing dangling rbd snapshots (for backups on cloudvirt1023) (T270478) [admin]
16:47 <dcaro> finished cleaning up the dangling snapshots from cloudvirt1024, freed ~12% of the capacity (T270478) [admin]
16:21 <dcaro> removing dangling rbd snapshots (for backups on cloudvirt1024) (T270478) [admin]
16:13 <andrewbogott> setting autoscale to 'off' for both ceph pools (eqiad1-compute and eqiad1-glance-images) because we like how things are set and the autoscaler does not [admin]
10:33 <dcaro> purging rbd snapshots for image fc6fb78b-4515-4dcc-8254-591b9fe01762 (T270478) [admin]
2020-12-17 §
22:17 <andrewbogott> correction to above, set the pg and pgp to 1024 for eqiad1-glance-images [admin]
22:16 <andrewbogott> setting pgp number to 8192 for eqiad1-compute (a 4x increase) and 2048 for eqiad1-glance-images (also a 4x increase) T270305 (same as pg) [admin]
22:14 <andrewbogott> setting pg number to 8192 for eqiad1-compute (a 4x increase) and 2048 for eqiad1-glance-images (also a 4x increase) T270305 [admin]
22:10 <andrewbogott> setting autoscale to 'warn' for both ceph pools (eqiad1-compute and eqiad1-glance-images) [admin]
2020-12-16 §
09:31 <dcaro> removing invalid backups from cloudvirt1024 (196 in total) (T269419) [admin]
2020-12-14 §
17:41 <dcaro> The removal freed ~12GB (still 100% usage :S) (T269419) [admin]
17:36 <dcaro> removing invalid backups that have a valid copy (T269419) [admin]
15:43 <dcaro> Merging the tagging for vm backups (T267195) [admin]
09:45 <arturo> icinga downtime cloudvirt1024 for 6 days (T269419) [admin]
2020-12-13 §
09:11 <_dcaro> running backup purge script on cloudvirt1024 (T269419) [admin]
2020-12-10 §
23:36 <bstorm> cleaned up the logs for haproxy on cloudcontrol1003 by deleting all the gzipped ones and truncating the .1 file [admin]
11:56 <dcaro> Freed some space on cloudvirt1024 by running the purge script (T269419) [admin]
09:17 <dcaro> removing leaked dns record discordwiki.eqiad.wmflabs (clinic duty) [admin]
2020-12-08 §
18:01 <dcaro> Host cloudvirt1030 up and running (T216195) [admin]
15:59 <dcaro> Re-imaging host cloudvirt1030 (T216195) [admin]
14:18 <dcaro> Host online cloudvirt1029 (T216195) [admin]
14:13 <dcaro> Host re-imaged, doing tests cloudvirt1029 (T216195) [admin]
12:14 <dcaro> Re-imaging cloudvirt1029 (T216195) [admin]
2020-12-07 §
18:33 <andrewbogott> putting cloudvirt1023 back into service T269467 [admin]
15:55 <andrewbogott> reimaging cloudvirt1028 for T216195 [admin]
14:49 <dcaro> Re-imaging cloudvirt1027 (T216195) [admin]