3451-3500 of 4724 results (28ms)
2021-11-04 §
16:16 <wm-bot> Finished rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by arturo@endurance [admin]
16:13 <wm-bot> Rebooting node cloudcephosd1022.eqiad.wmnet - cookbook ran by arturo@endurance [admin]
16:12 <wm-bot> Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance [admin]
16:12 <wm-bot> Adding new OSDs ['cloudcephosd1022.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance [admin]
16:00 <wm-bot> Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance [admin]
16:00 <wm-bot> Adding new OSDs ['cloudcephosd1022.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance [admin]
11:26 <wm-bot> Added 1 new OSDs ['cloudcephosd1021.eqiad.wmnet'] (T295012) - cookbook ran by arturo@endurance [admin]
11:26 <wm-bot> Added OSD cloudcephosd1021.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance [admin]
11:23 <wm-bot> Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by arturo@endurance [admin]
11:20 <wm-bot> Rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by arturo@endurance [admin]
11:19 <wm-bot> Adding OSD cloudcephosd1021.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance [admin]
11:19 <wm-bot> Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance [admin]
11:16 <wm-bot> Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance [admin]
2021-11-03 §
17:22 <arturo> [codfw1dev] installing keepalived 2.1.5 from buster-backports on cloudgw2001-dev/2002-dev (T294956) [admin]
11:45 <arturo> [codfw1dev] downgrade kernel on cloudgw2001-dev/2002-dev (T294853, T291813) [admin]
2021-11-02 §
10:54 <arturo> rebooting cloudnet1004/1003 for T291813 [admin]
10:43 <arturo> [codfw1dev] rebooting cloudgw200[12]-dev for T291813 [admin]
2021-10-24 §
00:47 <andrewbogott> deploying a change so that openstack clients use tls endpoints: https://gerrit.wikimedia.org/r/c/operations/puppet/+/732738 [admin]
2021-10-21 §
10:19 <arturo> drop firewall exception on core routers for wiki replicas legacy setup (T293897) [admin]
10:12 <arturo> drop NAT exception for wiki replicas legacy setup (T293897) [admin]
2021-10-20 §
21:06 <andrewbogott> creating cloudinfra-nfs project T293936 [admin]
2021-10-18 §
19:21 <andrewbogott> also ticked the 'admin' box on wikitech for majavah T292827 [admin]
18:58 <andrewbogott> granting majavah 'admin' role in the 'admin' project and also in the default domain. T292827 [admin]
2021-10-14 §
12:28 <arturo> [codfw1dev] add DB grants for cloudbackup2002.codfw.wmnet IP address to the cinder DB (T292546) [admin]
2021-10-13 §
10:46 <arturo> updating python3-neutron across the fleet (T292936) [admin]
2021-10-12 §
09:06 <dcaro> upgrading eqiad cloudnet hosts neutron packages (T292936) [admin]
08:57 <dcaro> upgrading codfw cloudnet hosts neutron packages (T292936) [admin]
2021-10-05 §
09:39 <arturo> [codfw1dev] cleaning up manila stuff from openstack (db, endpoints, tenant, VMs, and such) T291257 [admin]
2021-09-30 §
14:50 <andrewbogott> sudo cumin "cloud*" "ps -ef | grep nslcd && service nslcd restart" and sudo cumin "lab*" "ps -ef | grep nslcd && service nslcd restart" T292202 [admin]
14:43 <andrewbogott> ran sudo cumin --force --timeout 500 -o json "A:all" "ps -ef | grep nslcd && service nslcd restart" to get nslcd happy again T292202 [admin]
2021-09-29 §
09:41 <arturo> [codfw1dev] cleanup manila shares definitions for a clean start now that the manila-sharecontroller VM is apparently well configured (T291257) [admin]
2021-09-28 §
16:23 <bstorm> downtime for clouddb1020 to reduce re-pages in case this goes badly T291963 [admin]
16:21 <bstorm> powering on clouddb1020 via remote console T291963 [admin]
15:58 <bstorm> depooled clouddb1020 for repair T291961 [admin]
12:40 <dcaro> Merged change on sssd for bullseye cloud hosts (T291585) [admin]
11:30 <arturo> [codfw1dev] create floating IP 185.15.57.5 for manila-sharecontroller.cloudinfra-codfw1dev.codfw1dev.wmcloud.org (T291257) [admin]
2021-09-27 §
10:07 <arturo> cloudcontrol1004 apparently healthy T291446 [admin]
09:24 <arturo> rebooting cloudcontrol1004 for T291446 [admin]
2021-09-24 §
13:02 <arturo> [codfw1dev] create VM manila-share-controller-01 on cloudinfra-codfw1dev [admin]
13:00 <arturo> [codfw1dev] rebase labs/private.git on cloudinfra-puppetmaster-01, had merge conflict [admin]
2021-09-21 §
12:13 <arturo> [codfw1dev] trying to create a manila service image (T291257) [admin]
11:45 <arturo> [codfw1dev] created rabbitmq user (T291257) [admin]
11:32 <arturo> [codfw1dev] populated manila DB & created service endpoints (T291257) [admin]
11:06 <arturo> [codfw1dev] give manila user admin role @ manila project (T291257) [admin]
11:06 <arturo> [codfw1dev] created manila project (T291257) [admin]
10:57 <arturo> [codfw1dev] created manila user @ labtestwikitech (T291257) [admin]
10:49 <arturo> [codfw1dev] create manila database on cloudcontrol-dev nodes (galera) T291257 [admin]
2021-09-20 §
23:08 <bstorm> ran `echo check > /sys/block/md0/md/sync_action` on cloudcontrol1004 to check raid [admin]
22:48 <andrewbogott> stopped puppet & mariadb on cloudcontrol1004; it was flapping [admin]
22:44 <andrewbogott> sudo touch /tmp/galera.disabled on cloudcontrol1004, the service seems troubled there [admin]