3451-3500 of 4617 results (32ms)
2021-07-13 §
10:57 <dcaro> enabled autoscaling on codfw1 ceph cluster, setting a minimum of pgs on codfw1dev-compute to 128 [admin]
2021-07-02 §
10:12 <wm-bot> The cluster is not rebalance after adding the new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:12 <wm-bot> Added 2 new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:12 <wm-bot> Added OSD cloudcephosd1020.eqiad.wmnet... (2/2) (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:10 <wm-bot> Finished rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus [admin]
10:07 <wm-bot> Rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus [admin]
10:07 <wm-bot> Adding OSD cloudcephosd1020.eqiad.wmnet... (2/2) (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:07 <wm-bot> Added OSD cloudcephosd1019.eqiad.wmnet... (1/2) (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:05 <wm-bot> Finished rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus [admin]
10:02 <wm-bot> Rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus [admin]
10:01 <wm-bot> Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:01 <wm-bot> Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
09:13 <wm-bot> Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) (T285858) - cookbook ran by dcaro@vulcanus [admin]
09:13 <wm-bot> Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
2021-07-01 §
16:27 <bstorm> failed over cloudstore1009 to cloudstore1008 T224747 [admin]
16:18 <bstorm> downtimed cloudstore1008 and cloudstore1009 to fail over T224747 [admin]
14:25 <wm-bot> Adding OSD cloudcephosd1019.eqiad.wmnet... (2/3) (T285858) - cookbook ran by dcaro@vulcanus [admin]
14:25 <wm-bot> Added OSD cloudcephosd1017.eqiad.wmnet... (1/3) (T285858) - cookbook ran by dcaro@vulcanus [admin]
14:24 <wm-bot> Finished rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus [admin]
14:21 <wm-bot> Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus [admin]
14:20 <wm-bot> Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) (T285858) - cookbook ran by dcaro@vulcanus [admin]
14:20 <wm-bot> Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
14:18 <wm-bot> Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus [admin]
14:17 <wm-bot> Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) (T285858) - cookbook ran by dcaro@vulcanus [admin]
14:16 <wm-bot> Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
11:16 <wm-bot> Added new OSD node cloudcephosd1016.eqiad.wmnet (T285858) - cookbook ran by dcaro@vulcanus [admin]
11:13 <wm-bot> Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:58 <dcaro> rebooting cloudcephosd1016 (T285858) [admin]
10:47 <wm-bot> Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:44 <wm-bot> Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:42 <wm-bot> Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:41 <wm-bot> Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:40 <wm-bot> Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
2021-06-30 §
21:48 <bstorm> downtimed space alerts for scratch on cloudstore1008 until after the migration [admin]
2021-06-25 §
15:28 <andrewbogott> restarting openstack services on cloudcontrol1005 [admin]
09:16 <arturo> icinga downtime cloudcontrols for 2h [admin]
08:20 <dcaro> restarting rabbitmq on cloudcontrol100{3,4} [admin]
2021-06-21 §
13:54 <dcaro> puppet fix merged and deployed, servers are back to normal [admin]
13:20 <dcaro> merged broken puppet patch, downtimed all cloudvirts for 2h while fixing (nothing big, just added a bad systemd timer) [admin]
2021-06-20 §
22:21 <andrewbogott> clearing admin-monitoring VMs; puppet has been failing lately due to a full drive on the puppetmaster [admin]
2021-06-15 §
01:18 <bstorm> running a modified version of the prometheus dir size cron in screen T284964 [admin]
2021-06-14 §
10:13 <dcaro> setting ssd to debug mode on tools-sgeexec-0917 (T284130) [admin]
2021-06-10 §
10:58 <wm-bot> Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] (T281248) - cookbook ran by dcaro@vulcanus [admin]
10:58 <wm-bot> Finished rebooting node cloudcephmon2004-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [admin]
10:55 <wm-bot> Rebooting node cloudcephmon2004-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [admin]
10:55 <wm-bot> Finished rebooting node cloudcephmon2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [admin]
10:52 <wm-bot> Rebooting node cloudcephmon2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [admin]
10:52 <wm-bot> Finished rebooting node cloudcephmon2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [admin]
10:49 <wm-bot> Rebooting node cloudcephmon2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [admin]
10:49 <wm-bot> Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev (T281248) - cookbook ran by dcaro@vulcanus [admin]