1-50 of 1185 results (7ms)
2021-08-05 §
09:37 <dcaro> Taking one osd daemon down ot codfw cluster (T288203) [admin]
2021-08-04 §
19:20 <bd808> Running deleteBatch.php on cloudweb2001-dev to remove legacy Heira: pages from labtestwiki [admin]
2021-08-03 §
17:40 <bstorm> rerunning the glance backup script after failure [admin]
2021-07-31 §
00:10 <andrewbogott> "systemctl reset-failed cloud-init.service" on all VMs for T287309 [admin]
00:08 <andrewbogott> "systemctl reset-failed cloud-final.service" on all VMs for T287309 [admin]
2021-07-27 §
21:32 <andrewbogott> putting cloudvirt1012 back into service T286748 [admin]
20:52 <andrewbogott> draining VMs off of cloudvirt1012 so we can replace the battery for T286748 [admin]
15:15 <andrewbogott> "rm /etc/apt/sources.list.d/openstack-mitaka-jessie.list" cloud-wide [admin]
2021-07-23 §
15:22 <bstorm> update wikireplicas-dns for s7 fix for web replicas [admin]
2021-07-20 §
17:07 <andrewbogott> reloading haproxy on dbproxy1018 for T286598 [admin]
15:45 <arturo> failback from labstore1006 to labstore1007 (dumps NFS) https://gerrit.wikimedia.org/r/c/operations/puppet/+/705417 [admin]
00:10 <bstorm> restarting nova-api on cloudcontrol1003 to try and recover whatever it's doing with designate_floating_ip_ptr_records_updater [admin]
2021-07-19 §
22:05 <bstorm> set downtime scheduled for tomorrow from 1300 to 1600 UTC for cloudstore1008 and 1009 T286599 [admin]
20:40 <andrewbogott> reloading haproxy on dbproxy1018 for T286598 [admin]
13:50 <andrewbogott> upgrading mariadb to 10.3.29 on all cloudcontrols [admin]
2021-07-16 §
09:55 <dcaro> checking HP raid issues on coludvirt1012 (T286766) [admin]
2021-07-14 §
21:08 <andrewbogott> restarting lots of openstack services while trying to resolve T286675 [admin]
12:17 <dcaro> doing ceph outage tests on codfw1 (fyi) [admin]
2021-07-13 §
10:57 <dcaro> enabled autoscaling on codfw1 ceph cluster, setting a minimum of pgs on codfw1dev-compute to 128 [admin]
2021-07-02 §
10:12 <wm-bot> The cluster is not rebalance after adding the new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:12 <wm-bot> Added 2 new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:12 <wm-bot> Added OSD cloudcephosd1020.eqiad.wmnet... (2/2) (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:10 <wm-bot> Finished rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus [admin]
10:07 <wm-bot> Rebooting node cloudcephosd1020.eqiad.wmnet - cookbook ran by dcaro@vulcanus [admin]
10:07 <wm-bot> Adding OSD cloudcephosd1020.eqiad.wmnet... (2/2) (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:07 <wm-bot> Added OSD cloudcephosd1019.eqiad.wmnet... (1/2) (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:05 <wm-bot> Finished rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus [admin]
10:02 <wm-bot> Rebooting node cloudcephosd1019.eqiad.wmnet - cookbook ran by dcaro@vulcanus [admin]
10:01 <wm-bot> Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:01 <wm-bot> Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
09:13 <wm-bot> Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) (T285858) - cookbook ran by dcaro@vulcanus [admin]
09:13 <wm-bot> Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
2021-07-01 §
16:27 <bstorm> failed over cloudstore1009 to cloudstore1008 T224747 [admin]
16:18 <bstorm> downtimed cloudstore1008 and cloudstore1009 to fail over T224747 [admin]
14:25 <wm-bot> Adding OSD cloudcephosd1019.eqiad.wmnet... (2/3) (T285858) - cookbook ran by dcaro@vulcanus [admin]
14:25 <wm-bot> Added OSD cloudcephosd1017.eqiad.wmnet... (1/3) (T285858) - cookbook ran by dcaro@vulcanus [admin]
14:24 <wm-bot> Finished rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus [admin]
14:21 <wm-bot> Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus [admin]
14:20 <wm-bot> Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) (T285858) - cookbook ran by dcaro@vulcanus [admin]
14:20 <wm-bot> Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
14:18 <wm-bot> Rebooting node cloudcephosd1017.eqiad.wmnet - cookbook ran by dcaro@vulcanus [admin]
14:17 <wm-bot> Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) (T285858) - cookbook ran by dcaro@vulcanus [admin]
14:16 <wm-bot> Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
11:16 <wm-bot> Added new OSD node cloudcephosd1016.eqiad.wmnet (T285858) - cookbook ran by dcaro@vulcanus [admin]
11:13 <wm-bot> Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:58 <dcaro> rebooting cloudcephosd1016 (T285858) [admin]
10:47 <wm-bot> Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:44 <wm-bot> Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:42 <wm-bot> Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]
10:41 <wm-bot> Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus [admin]