5201-5250 of 5605 results (30ms)
2020-05-27 §
23:29 <andrewbogott> disabling the backup job on cloudbackup2001 (just like last week) so the backup doesn't start while Brooke is rebuilding labstore1004 tomorrow. [admin]
06:03 <bd808> `systemctl start mariadb` on clouddb1001 following reboot (take 2) [admin]
05:58 <bd808> `systemctl start mariadb` on clouddb1001 following reboot [admin]
05:53 <bd808> Hard reboot of clouddb1001 via Horizon. Console unresponsive. [admin]
2020-05-25 §
16:35 <arturo> [codfw1dev] created zone `0-29.57.15.185.in-addr.arpa.` (T247972) [admin]
2020-05-21 §
19:23 <andrewbogott> disabling puppet on cloudbackup2001 to prevent the backup job from starting during maintenance [admin]
19:16 <andrewbogott> systemctl disable block_sync-tools-project.service on cloudbackup2001.codfw.wmnet to avoid stepping on current upgrade [admin]
15:48 <andrewbogott> re-imaging cloudnet1003 with Buster [admin]
2020-05-19 §
22:59 <bd808> `apt-get install mariadb-client` on cloudcontrol1003 [admin]
21:12 <bd808> Migrating wcdo.wcdo.eqiad.wmflabs to cloudvirt1023 (T251065) [admin]
2020-05-18 §
21:37 <andrewbogott> rebuilding cloudnet2003-dev with Buster [admin]
2020-05-15 §
22:10 <bd808> Added reedy as projectadmin in cloudinfra project (T249774) [admin]
22:05 <bd808> Added reedy as projectadmin in admin project (T249774) [admin]
18:44 <bstorm_> rebooting cloudvirt-wdqs1003 T252831 [admin]
15:47 <bd808> Manually running wmcs-novastats-dnsleaks from cloudcontrol1003 (T252889) [admin]
2020-05-14 §
23:28 <bstorm_> downtimed cloudvirt1004/6 and cloudvirt-wdqs1003 until tomorrow around this time T252831 [admin]
22:21 <bstorm_> upgrading qemu-system-x86 on cloudvirt1006 to backports version T252831 [admin]
22:15 <bstorm_> changing /etc/libvirt/qemu.conf and restarting libvirtd on cloudvirt1006 T252831 [admin]
21:12 <andrewbogott> rebuilding cloudvirt1003-wdqs as part of T252831 [admin]
15:47 <andrewbogott> moving cloudvirt1004 and cloudvirt1006 to the 'ceph' aggregate for T252784 [admin]
15:02 <andrewbogott> moving all of cloudvirt100[1-9] into the 'toobusy' host aggregate. These are slower, have spinning disks, and are due for replacement. [admin]
2020-05-12 §
20:33 <andrewbogott> moving cloudvirt1023 to the 'standard' pool and out of the 'spare' pool [admin]
19:10 <jeh> disable neutron-openvswitch-agent service on cloudvirt2001-dev.codfw T248881 [admin]
19:09 <jeh> Shutdown the unused eno2 network interface on cloudvirt2001-dev.codfw to clear up monitoring errors T248425 [admin]
18:20 <andrewbogott> moving cloudvirt1024 out of the 'maintenance' aggregate and into 'spare' [admin]
16:45 <andrewbogott> restarting neutron-l3-agent on cloudnet1004 so it knows about all three cloudcontrols. Leaving cloudnet1003 since restarting it there will cause network interruptions [admin]
14:06 <arturo> icinga downtime everything for 2h for Debian Buster migration in some cloud components [admin]
2020-05-09 §
16:53 <andrewbogott> rebuilding cloudcontrol2001-dev and 2003-dev with buster for T252121 [admin]
2020-05-08 §
19:02 <bstorm_> moving tools-k8s-haproxy-2 from cloudvirt1021 to cloudvirt1017 to improve spread [admin]
2020-05-05 §
13:58 <andrewbogott> rebuilding cloudcontrol2004-dev to test new puppet changes [admin]
2020-05-04 §
09:04 <arturo> [codfw1dev] manually modify iptables ruleset to only allow SSH from WMF bastions on cloudservices2003-dev and cloudcontrol2004-dev (T251604) [admin]
2020-04-21 §
22:12 <andrewbogott> moving cloudvirt1004 out of the 'standard' aggregate and into the 'maintenance' aggregate [admin]
16:01 <jeh> restart cloudceph mon and osd services for openssl upgrades [admin]
2020-04-15 §
18:44 <jeh> create indexes and views for grwikimedia T245912 [admin]
2020-04-13 §
15:07 <jeh> restart memcached on labwebs to increase cache size T145703 [admin]
2020-04-09 §
19:57 <andrewbogott> upgrading eqiad1 designate to rocky [admin]
16:52 <andrewbogott> cleaned up a bunch of leaked .eqiad.wmflabs dns records [admin]
2020-04-08 §
19:20 <andrewbogott> rotated password and api token for pdns servers on cloudservices1003 and cloudservices1004 [admin]
14:54 <arturo> `root@cloudcontrol1003:~# cp /etc/inputrc .inputrc` to solve some bash shortcut weirdness [admin]
2020-04-07 §
20:57 <andrewbogott> service sssd stop; rm -rf /var/lib/sss/db*; service sssd start on tools-sgebastion-08 [admin]
2020-04-06 §
22:39 <andrewbogott> deleting bogus groups cn=b'project-bastion',ou=groups,dc=wikimedia,dc=org and cn=b'project-tools',ou=groups,dc=wikimedia,dc=org from ldap [admin]
17:42 <arturo> [codfw1dev] transferred DNS zone 57.15.185.in-addr.arpa. to the cloudinfra-codfw1dev project (T247972) [admin]
17:39 <arturo> [codfw1dev] `openstack zone create --email root@wmflabs.org --type PRIMARY --ttl 3600 --description "floating IPs subnet" 57.15.185.in-addr.arpa.` (T247972) [admin]
16:23 <arturo> restarting apache2 in cloudcontrol1003/1004 to pick up latest wmfkeystonehooks changes T249494 [admin]
2020-04-02 §
20:59 <jeh> codfw1dev clear VM error states and start bastions, puppet master and database [admin]
20:53 <jeh> codfw1dev clear VM error states and start bastions, puppet master and database [admin]
2020-04-01 §
16:27 <arturo> [codfw1dev] enable puppet across the fleet clean vxlan changes (T248881) [admin]
2020-03-31 §
12:35 <arturo> [codfw1dev] restarting VMs: designaterockytest14, bastion-codfw1dev-0[1,2] (T248881) [admin]
12:34 <arturo> [codfw1dev] installing neutron-openvswitch-agent on cloudvirt2001-dev (T248881) [admin]
12:25 <arturo> [codfw1dev] installing neutron-openvswitch-agent on cloudnet200[2,3]-dev (T248881) [admin]