1151-1200 of 3677 results (23ms)
2022-08-11 §
08:36 <wm-bot2> Rebooting node cloudcephosd1025.eqiad.wmnet (T314870) - cookbook ran by fran@MacBook-Pro.station [admin]
08:36 <wm-bot2> Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@MacBook-Pro.station [admin]
08:36 <wm-bot2> Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@MacBook-Pro.station [admin]
2022-08-10 §
13:10 <wm-bot2> Finished rebooting node cloudcephosd1025.eqiad.wmnet (T314870) - cookbook ran by fran@MacBook-Pro.station [admin]
13:06 <wm-bot2> Rebooting node cloudcephosd1025.eqiad.wmnet (T314870) - cookbook ran by fran@MacBook-Pro.station [admin]
13:06 <wm-bot2> Adding OSD cloudcephosd1025.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@MacBook-Pro.station [admin]
13:06 <wm-bot2> Adding new OSDs ['cloudcephosd1025.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@MacBook-Pro.station [admin]
2022-08-04 §
17:16 <taavi> deleted all scheduler_fanout_ rabbit queues in an attempt to fix scheduling [admin]
16:32 <taavi> restart neutron-l3-agent to pick up rabbit config changes [admin]
15:12 <andrewbogott> stopping rabbitmq on cloudcontrol1xxx [admin]
09:57 <taavi> stop wikitech_run_jobs.timer on labweb1001/1002, hosts pending decom [admin]
2022-08-03 §
20:55 <andrewbogott> root@tools-checker-04:~# systemctl restart uwsgi-toolschecker_cron.service [admin]
20:41 <andrewbogott> restarting neutron-l3-agent.service on cloudnet1003 and 1004. The agent was routing properly but had lost touch with rabbitmq [admin]
2022-08-02 §
14:07 <andrewbogott> shutting down codfw1dev ceph cluster according to https://docs.mirantis.com/mcp/q4-18/mcp-operations-guide/scheduled-maintenance-power-outage/power-off-ceph-cluster.html [admin]
13:54 <andrewbogott> shutting down basically all of codfw1dev to support pdu maintenance -- all the ceph OSDs will lose power so best to have everything stopped. [admin]
2022-07-27 §
19:32 <andrewbogott> switching the openstack.eqiad1.wikimedia.cloud endpoint from cloudcontrol1004 to 1006, https://gerrit.wikimedia.org/r/c/operations/dns/+/817878/2/templates/wikimediacloud.org#54 [admin]
16:33 <andrewbogott> here is a test message in the admin channel [admin]
2022-07-25 §
13:43 <andrewbogott> pooling cloudweb100[34] and depooling labweb100[12] for testing in prep for decomming labweb100[12] [admin]
2022-07-22 §
16:41 <taavi> depool cloudweb1003/1004 since horizon seems to be having issues [admin]
16:22 <taavi> pooling cloudweb1003/1004 now that grant issues are sorted [admin]
2022-07-21 §
18:26 <andrewbogott> depooling cloudweb1003 and 1004 for wikitech, horizon, striker -- pending db grant changes [admin]
18:06 <andrewbogott> pooling cloudweb1003 and 1004 for wikitech, horizon, striker [admin]
2022-07-20 §
18:02 <dcaro> things seem stable, trying to bring up a the last rabbit node, cloudcontrol1007 (T313400) [admin]
17:45 <bd808> `sudo service striker restart` on labweb1002 [admin]
17:43 <bd808> `sudo service striker restart` on labweb1001 [admin]
17:10 <dcaro> things seem stable, trying to bring up a fourth rabbit node, cloudcontrol1006 (T313400) [admin]
16:26 <dcaro> things seem stable, trying to bring up a third, cloudcontrol1005 (T313400) [admin]
15:51 <dcaro> things seem stable now with one rabbit node, trying to bring up a second (T313400) [admin]
14:16 <dcaro> stopping rabbin on cloudcontrol1004, leaving only 1003 alive (T313400) [admin]
13:17 <dcaro> restarting the whole rabbit cluster (T313400) [admin]
2022-07-19 §
16:30 <wm-bot2> Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster [admin]
16:30 <wm-bot2> Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster [admin]
16:26 <wm-bot2> Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@buster [admin]
16:18 <wm-bot2> Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster [admin]
16:18 <wm-bot2> Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster [admin]
16:14 <wm-bot2> Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster [admin]
16:01 <wm-bot2> Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster [admin]
16:01 <wm-bot2> Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster [admin]
15:57 <wm-bot2> Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster [admin]
15:57 <wm-bot2> Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: 3da2d4f6-5c5b-4a21-9a0b-2b010960ed6a, use this to unset). - cookbook ran by andrew@buster [admin]
15:56 <wm-bot2> Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster [admin]
15:56 <wm-bot2> Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@buster [admin]
15:56 <wm-bot2> Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster [admin]
15:56 <wm-bot2> Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster [admin]
15:52 <wm-bot2> Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@buster [admin]
15:49 <wm-bot2> Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: 8a10505a-107d-4e78-ba84-363a7dea8d69, use this to unset). - cookbook ran by andrew@buster [admin]
15:47 <wm-bot2> Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 752aa58b-44d4-4340-b05d-911b14f2314d, use this to unset). - cookbook ran by andrew@buster [admin]
15:47 <wm-bot2> Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: f89f0851-ce3f-4a92-899e-0d4638acc9c3, use this to unset). - cookbook ran by andrew@buster [admin]
15:46 <wm-bot2> Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster [admin]
15:46 <wm-bot2> Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@buster [admin]