2301-2350 of 3677 results (21ms)
2022-03-08 §
13:07 <arturo> [codfw1dev] rebooting cloudvirt servers for T303179 [admin]
13:06 <arturo> [codfw1dev] rebooting cloudnet servers for T303179 [admin]
12:55 <arturo> [codfw1dev] rebooting cloudcontrol servers for T303179 [admin]
2022-03-03 §
08:49 <taavi> deploying cloudmetrics grafana to grafana 8, T282863 [admin]
2022-03-02 §
09:06 <arturo> merging core router firewall change https://gerrit.wikimedia.org/r/c/operations/homer/public/+/701347 [admin]
2022-02-28 §
15:30 <dcaro> cleaning up leftover snapshots from failed backups of the maps volume (T302720) [admin]
2022-02-24 §
17:04 <andrewbogott> upgrading eqiad1 and codfw1dev to mariadb 10.5.15+maria~bullseye via 'apt-get install libmariadb3:amd64 galera-4 mariadb-server' [admin]
15:42 <dcaro> stopping and starting mariadb on cloudcontrol1003 (T302146) [admin]
10:37 <arturo> [codfw1dev] briefly installed galera-4 (26.4.11+1bullseye) over (26.4.9-0+deb11u1) on cloudcontrol2001-dev and then downgrade again to verify package install (T302482) [admin]
2022-02-23 §
20:39 <taavi> added domain-wide 'designateadmin' and 'observer' roles to project-proxy-dns-manager service account T295246 [admin]
17:40 <andrewbogott> restarting lots of openstack services to try to clear up the mess that is T236101 [admin]
12:13 <arturo> cleaning up cinder volume snapshots, aborrero@cloudcontrol1005:~$ for i in $(sudo wmcs-openstack volume snapshot list -f value -c ID) ; do sudo wmcs-openstack volume snapshot delete $i ; done (T302382) [admin]
10:14 <arturo> cleaning up neutron agents for non-existent servers cloudvirt100[1-9].eqiad.wmnet,cloudvirt10[12-15].eqiad.wmnet [admin]
10:05 <dcaro> Deleting stuck novafullstack servers, to let the service create new ones (T302369) [admin]
09:56 <arturo> neutron agent-delete bad663b3-fd25-4393-a546-4b1b4bdec4db (Linux bridge agent | cloudvirtan1001) [admin]
09:56 <arturo> neutron agent-delete 1071c198-ed57-4b5a-9439-30e66a31aa69 (Linux bridge agent | cloudvirtan1005) [admin]
09:55 <arturo> neutron agent-delete 2eeef198-8af7-4e5d-bd73-e14a2a8d2404 (Linux bridge agent | cloudvirtan1004) [admin]
09:55 <arturo> neutron agent-delete afe173eb-35ba-444a-9960-899629786d2f (Linux bridge agent | cloudvirtan1003) [admin]
09:54 <arturo> neutron agent-delete afcb9b7f-c1a6-4ff4-9b10-92bfbe8d1a56 (Linux bridge agent | cloudvirtan1002) [admin]
09:39 <dcaro> restarting neutron-api cloudcontrol1003 to see if the agent status update starts working (T302369) [admin]
09:38 <dcaro> restarting neutron-dhcp-agent on cloudnet1003 (T302369) [admin]
2022-02-22 §
22:10 <andrewbogott> raising project 'maps' quota by two tb -- T300160 [admin]
09:24 <arturo> restarting mariadb @ cloudcontrol1003 (T302146) [admin]
09:13 <arturo> restarting mariadb @ cloudcontrol1004 (T302146) [admin]
2022-02-18 §
21:57 <andrewbogott> leaving cloudcontrol1003 downtimed with disabled puppet for the weekend. Everything there should be stable and fine save rabbit which needs an upgrade. [admin]
21:30 <andrewbogott> rebooting cloudcontrol1003 because rabbit is freaking out [admin]
17:25 <andrewbogott> in-place upgrade of cloudcontrol1004 to bullseye -- T281276 [admin]
12:34 <arturo> manually install prometheus-openstack-exporter on cloudcontrol1005 (T302050) [admin]
2022-02-17 §
23:02 <andrewbogott> in-place upgrade to Bullseye on cloudcontrol1005 T281276 [admin]
2022-02-15 §
14:15 <taavi> [codfw1dev] added domain-wide 'designateadmin' and 'observer' roles to codfw1dev-proxy-dns-manager service account T295246 [admin]
2022-02-04 §
10:12 <arturo> restart backup_vms service in cloudvirt1024 (T300956) [admin]
2022-02-03 §
08:21 <taavi> cloudmetrics1004: manually added an empty line to /etc/prometheus/blackbox.yml to make /usr/local/bin/blackbox-exporter-assemble happy (clearing "performing a change every puppet run" alert) [admin]
2022-02-02 §
02:36 <andrewbogott> restarting mariadb on cloudcontrol1004 [admin]
2022-01-31 §
10:15 <arturo> cloudcontrol1005:~$ sudo systemctl restart backup_glance_images.service (failed state, no logs, icinga alert) [admin]
2022-01-29 §
18:24 <taavi> delete 2 puppet prefixes in a weird state T299750 [admin]
2022-01-27 §
13:24 <arturo> cloudmetrics1004:~ $ sudo systemctl restart wmcs_monitoring_graphite_rsync.service (T300138) [admin]
2022-01-26 §
19:09 <andrewbogott> bootstrapping a fresh galera node on cloudcontrol1004 [admin]
18:57 <andrewbogott> restarting mariadb on cloudcontrol1004 [admin]
2022-01-25 §
10:49 <arturo> made cloudmetrics1001/1002 primary/backup respectively (T299744, T297814, T300011) [admin]
2022-01-19 §
16:38 <andrewbogott> moving all scratch mounts to scratch.svc.cloudinfra-nfs.eqiad1.wikimedia.cloud [admin]
2022-01-05 §
03:11 <andrewbogott> 'cp /etc/apt/sources.list /etc/apt/sources.list.prepuppet' on all VMs. Backing up state before puppetizing sources.list with https://gerrit.wikimedia.org/r/c/operations/puppet/+/751498 [admin]
2022-01-04 §
12:44 <dcaro> increasing the size_limit for labs ldap servers [admin]
2021-12-26 §
16:55 <majavah> run attachLdapUser.php on wikitech for developer account "Karthiksripal" [admin]
2021-12-24 §
22:51 <majavah> ran the wikireplica dns script on s5 T298303 [admin]
2021-12-23 §
21:42 <majavah> deployed horizon wmf-proxy-dashboard update to fix editing of existing proxies [admin]
2021-12-21 §
10:39 <arturo> dropped egress NAT exceptions for WMF apt repos, T298042 [admin]
2021-12-15 §
12:44 <dcaro> Downtiming cloudvirt-wdqs1001 as it has no VMs running until disk space is fixed (T297454) [admin]
2021-12-14 §
10:26 <dcaro> Moved the nova cache (/var/lib/nova/instances/_base) and the canary image local data (/var/lib/nova/instance/<canary_image_id>) to the root disk on cloudvirt-wdqs1001 to temporary free some space (T297454) [admin]
2021-12-13 §
18:08 <wm-bot> Drained 'cloudvirt1014.eqiad.wmnet'. - cookbook ran by michael@mouse [admin]
17:50 <wm-bot> Set cloudvirt 'cloudvirt1014.eqiad.wmnet' maintenance. - cookbook ran by michael@mouse [admin]