1-50 of 1084 results (19ms)
2021-06-08 §
23:19 <bd808> Downtimed cloudmetrics1002 in icinga until 2021-06-30 23:59:01 (T281881) [admin]
21:08 <bstorm> downtiming grafana-labs for maintenance [admin]
16:28 <wm-bot> Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus [admin]
16:27 <wm-bot> Finished rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [admin]
16:24 <wm-bot> Rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [admin]
16:24 <wm-bot> Finished rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [admin]
16:22 <wm-bot> Rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [admin]
16:21 <wm-bot> Finished rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [admin]
16:18 <wm-bot> Rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [admin]
16:18 <wm-bot> Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus [admin]
16:17 <wm-bot> Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus [admin]
15:03 <wm-bot> Finished rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
14:59 <wm-bot> Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
14:59 <wm-bot> Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
14:57 <wm-bot> Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
14:57 <wm-bot> Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
14:29 <wm-bot> Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
14:23 <wm-bot> Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
14:18 <wm-bot> Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [admin]
2021-06-07 §
14:26 <andrewbogott> moving cloudvirt1040 from 'maintenance' aggregate to 'ceph' aggregate T281399 [admin]
2021-06-01 §
13:12 <dcaro> Changed the ceph osd_memory_target on eqiad pool to 6Gi (we were reaching the limit, swapping at some points) [admin]
09:57 <arturo> fix PTR record for 185.15.56.1 (T284025) [admin]
09:56 <arturo> fix PTR record for 185.15.56.1 (T248025) [admin]
2021-05-27 §
14:58 <wm-bot> Testing - cookbook ran by dcaro@vulcanus [admin]
2021-05-26 §
19:10 <andrewbogott> reimaging cloudvirt1018 to support local VM storage [admin]
18:07 <andrewbogott> draining cloudvirt1018, converting it to a local-storage host like cloudvirt1019 and 1020 -- T283296 [admin]
14:36 <dcaro> Enabled syslog logging for osd.55 on eqiad ceph cluster for testing (T281247) [admin]
14:36 <dcaro> Enabled syslog logging on codfw ceph cluster (mon/osd/mgr) (T281247) [admin]
11:26 <arturo> [codfw1dev] purge old kernel packages in cloudvirt200[12]-dev [admin]
11:03 <arturo> created public flavor `g3.cores16.ram36.disk20` (even though it was requested as private in T283293, but may be useful for others) [admin]
2021-05-25 §
16:14 <bd808> Closed #wikimedia-cloud-admin on f***node [admin]
16:11 <bd808> Closed #wikimedia-cloud-feed on f***node [admin]
15:19 <dcaro> rebooted cloudvirt1020, starting VMs (T275893) [admin]
15:13 <dcaro> rebooting cloudvirt1020 (T275893) [admin]
14:42 <dcaro> taking cloudvirt1020 out for maintenance (openstack wise) so no new VMs are scheduled on it (T275893) [admin]
2021-05-24 §
22:32 <andrewbogott> changing the default ttl for eqiad1.wikimedia.cloud. from 3600 to 60; this should help us avoid madness when re-using hostnames. [admin]
11:20 <arturo> created `g3.cores2.ram80.disk40.private` for the wmf-research-tools project, to allow resizing a 40G disk instance [admin]
2021-05-22 §
02:14 <bstorm> downtiming SMART alerts on dumps server labstore1007 for the weekend because it has been flapping T281045 [admin]
2021-05-13 §
21:25 <bstorm> converted the maps and scratch volumes on cloudstore1008 (standby) to drbd T224747 [admin]
15:45 <bstorm> re-running wikireplicas-dns after refactor of config to make sure it doesn't change anything [admin]
2021-05-12 §
14:22 <arturo> [codfw1dev] cleanup old unused agents (bgp, ovs) [admin]
11:37 <arturo> [codfw1dev] replacing cloudnet2003-dev with cloudnet2004-dev (T281381) [admin]
2021-05-11 §
18:00 <andrewbogott> adding 'trove' service project in advance of deploying trove in eqiad1 [admin]
10:22 <arturo> rebooted cloudgw1002 (active) thus causing a failover to cloudgw1001 [admin]
2021-05-09 §
10:53 <arturo> icinga-downtime cloudmetrics1002 for 3 months (T275605) [admin]
2021-05-07 §
13:51 <andrewbogott> add inherited 'admin' right to novaadmin user throughout eqiad1. I was trying to narrow down the rights here but lack of admin breaks some workflows, e.g. T281894 and T282235 [admin]
2021-05-06 §
15:31 <arturo> about to migrating CloudVPS network to the cloudgw architecture T270704 [admin]
11:14 <dcaro> restarting cinder-volume on the eqiad control nodes to refresh the ceph libraries (T282109) [admin]
2021-05-05 §
16:07 <dcaro> disallowing insecure global ids on the eqiad ceph cluster (T280641) [admin]
15:15 <wm-bot> Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus [admin]