1701-1750 of 1972 results (27ms)
2020-06-11 §
12:39 <arturo> for the record, k8s etcd servers certificate changed (puppet based) and k8s just kept working [toolsbeta]
12:35 <arturo> according to `aborrero@cloud-cumin-01:~$ sudo cumin --force -x 'O{project:toolsbeta}' 'run-puppet-agent'` we are mostly back in business [toolsbeta]
12:14 <arturo> try switching all VMs to toolsbeta-puppetmaster-04 [toolsbeta]
12:14 <arturo> poweroff toolsbeta-puppetmaster-03 [toolsbeta]
12:12 <arturo> copy over labs/private from toolsbeta-puppetmaster-03 to toolsbeta-puppetmaster-04 [toolsbeta]
11:53 <arturo> create VM toolsbeta-puppetmaster-04 [toolsbeta]
11:35 <arturo> try reinstalling the python3 stack in toolsbeta-puppetmaster-03, because everything python-related segfaults [toolsbeta]
11:33 <arturo> reboot toolsbeta-puppetmaster-03 to try cleaning up potential kernel/filesystem problems [toolsbeta]
11:32 <arturo> apparently every python script segfaults in toolsbeta-puppetmaster-03 [toolsbeta]
11:27 <arturo> puppetdb wasn't the problem. The problem is puppet-enc segfaulting in toolsbeta-puppetmaster-03 [toolsbeta]
11:21 <arturo> puppet not working bc puppetdb, run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` [toolsbeta]
2020-06-04 §
21:06 <andrewbogott> added krenair to toolsbeta.admin group in ldap [toolsbeta]
2020-05-28 §
11:27 <arturo> cleanup livehackings [toolsbeta]
10:31 <arturo> livehacking puppetmaster and toolsbeta-proxy-1 to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 (T253816) [toolsbeta]
10:30 <arturo> livehacking puppetmaster to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 [toolsbeta]
2020-05-27 §
12:02 <arturo> the k8s cluster is now running v1.16.10 (T246122) [toolsbeta]
11:05 <arturo> trying `modules/kubeadm/files/wmcs-k8s-node-upgrade.py --control toolsbeta-test-k8s-control-1 --project toolsbeta --domain eqiad.wmflabs --src-version 1.15 --dst-version 1.16.10 -n toolsbeta-test-k8s-worker-1 -n toolsbeta-test-k8s-worker-2 -n toolsbeta-test-k8s-worker-3` (T246122) [toolsbeta]
11:02 <arturo> upgraded the rest of the k8s control plane nodes to 1.16.10 (T246122) [toolsbeta]
10:58 <arturo> running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo apt-get install kubelet -y` in the 1.16 version from the component repo (T246122) [toolsbeta]
10:58 <arturo> running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` and this time it works! (T246122) [toolsbeta]
2020-05-26 §
16:17 <bstorm_> fix incorrect volume name in kubeadm-config T246122 [toolsbeta]
15:02 <arturo> first k8s upgrade failed for yet-to-be-known reasons (T246122) [toolsbeta]
14:54 <arturo> `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` (T246122) [toolsbeta]
14:54 <arturo> bump installed version of kubeadm and kubectl to 1.16.10 (T246122) [toolsbeta]
09:57 <arturo> installing kubectl/kubeadm 1.16.9 on k8s worker nodes (T246122) [toolsbeta]
09:56 <arturo> installing kubectl/kubeadm 1.16.9 on k8s control nodes (T246122) [toolsbeta]
09:30 <arturo> set `profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-16'` at project level for trying T246122 [toolsbeta]
09:25 <arturo> `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` broken puppet in this project because puppetdb is down again [toolsbeta]
2020-05-21 §
22:14 <bd808> Building tools-webservice 0.70 via wmcs-package-build.py [toolsbeta]
2020-05-19 §
12:20 <arturo> trying to install tesseract 4.1.0 in toolsbeta-sgebastion-04 (T247422) [toolsbeta]
10:18 <arturo> `aborrero@toolsbeta-puppetdb-02:~$ sudo systemctl restart puppetdb` [toolsbeta]
2020-05-15 §
20:48 <bstorm_> found an error in the new version of maintain-kubeusers, removing the deployment for now T246059 [toolsbeta]
20:35 <bstorm_> updating the maintain-kubeusers image to be able to control admin accounts [toolsbeta]
2020-05-14 §
12:09 <arturo> created puppet prefix toolsbeta-acme-chief in horizon (T252762) [toolsbeta]
12:08 <arturo> created toolsbeta-acme-chief-01 VM (T252762) [toolsbeta]
2020-05-12 §
18:35 <bstorm_> upgraded to using typha and rolled back to not doing so -- no affect on existing network T250863 [toolsbeta]
17:44 <bstorm_> set the calico version to v3.14.0 because the new liveness probe isn't compatible with the old version. T250863 [toolsbeta]
17:35 <bstorm_> deployed an updated bit of yaml for calico without upgrading the version first T250863 [toolsbeta]
2020-05-08 §
12:48 <arturo> allocated floating IP `185.15.56.12` for the VM `toolsbeta-email-01` and FQDN `mail.toolsbeta.wmflabs.org` (T120225) [toolsbeta]
12:24 <arturo> added puppet prefix `toolsbeta-email` (T120225) [toolsbeta]
2020-05-07 §
16:33 <arturo> livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594945 (T251297 and T250866) [toolsbeta]
12:36 <arturo> cleanup livehacks in toolsbeta-puppetmaster-03 [toolsbeta]
11:12 <arturo> livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594925 and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594926 (T251297 and T250866) [toolsbeta]
2020-05-06 §
19:11 <bstorm_> updated toollabs-webservice to 0.69 for toolsbeta [toolsbeta]
09:58 <arturo> livehacking toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594471 (T251297) [toolsbeta]
2020-05-05 §
10:04 <arturo> add herron as user and projectadmin, we will work on the email setup (T120225) [toolsbeta]
09:59 <arturo> created VM toolsbeta-mail-01 (T120225) [toolsbeta]
2020-05-04 §
13:02 <arturo> `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb.service` trying to bring back puppetdb, which is preventing puppet agent runs in the whole project [toolsbeta]
2020-04-29 §
19:48 <bstorm_> ran the scary rewrite-psp-preset.sh script across toolsbeta T247455 [toolsbeta]
2020-04-20 §
14:47 <arturo> added joakino to toolsbeta.admin LDAP group [toolsbeta]