1201-1250 of 1498 results (30ms)
2020-09-09 §
11:42 <arturo> actually, the whole k8s cluster seems down? the API seems down at least [toolsbeta]
11:39 <arturo> all 3 k8s control nodes seem in bad shape. Wont let me ssh in, or use the console access. Try force-rebooting them [toolsbeta]
11:27 <arturo> created 2 VMs: toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2 (T250172) [toolsbeta]
11:25 <arturo> created new server group toolsbeta-k8s-ingress (T250172) [toolsbeta]
11:24 <arturo> created new puppet prefix `toolsbeta-test-k8s-ingress` (T250172) [toolsbeta]
2020-07-15 §
21:35 <bstorm> set all of toolsbeta to mount NFS 4.2 except the bastion T257945 [toolsbeta]
2020-07-14 §
22:28 <bstorm> rebooting toolsbeta-sgebastion-04 during NFS testing thing [toolsbeta]
2020-07-08 §
11:08 <arturo> live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/610029 (T234617) [toolsbeta]
2020-06-26 §
12:12 <arturo> puppetmaster live-hacking with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/608005 (T120210) [toolsbeta]
2020-06-24 §
12:55 <arturo> live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607279 (T120225) [toolsbeta]
12:23 <arturo> live-hacking puppetmaster with exim prometheus stuff (T175964) [toolsbeta]
11:31 <arturo> live-hack the puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607320 (T175964) [toolsbeta]
11:26 <arturo> add TXT record `"v=spf1 mx -all"` T120225 [toolsbeta]
11:24 <arturo> fix MX record for toolsbeta.wmflabs.org (missing trailing dot) T120225 [toolsbeta]
2020-06-23 §
13:10 <arturo> added herron to the test tool for email testing [toolsbeta]
11:36 <arturo> removing `benapetr` and adding myself to the test tool [toolsbeta]
11:02 <arturo> setting `profile::toolforge::mail_domain: toolsbeta.wmflabs.org` in toolsbeta-mail puppet prefix [toolsbeta]
10:55 <arturo> allow ingress smtp/smtps traffic in the MTA security group [toolsbeta]
10:52 <arturo> created MX record pointing to mail.toolsbeta.wmflabs.org [toolsbeta]
09:43 <arturo> restarted nginx in toolsbeta-acme-chief-01 to pickup new certificate, otherwise clients won't accept its TLS cert [toolsbeta]
09:38 <arturo> live-hacking toolsbeta-puppetmaster-04 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/607251 [toolsbeta]
2020-06-16 §
22:54 <bd808> Building webservice 0.72 [toolsbeta]
2020-06-15 §
21:54 <bstorm_> removed killgridjobs.sh from toolsbeta bastion T157792 [toolsbeta]
17:52 <bd808> Building webservice 0.71 [toolsbeta]
2020-06-12 §
19:41 <bstorm_> set `profile::wmcs::nfsclient::mode: soft` on toolsbeta-workflow-test T127559 [toolsbeta]
2020-06-11 §
12:42 <arturo> introduce puppet profile 'toolsbeta-docker-registry' and relocate some hiera config there [toolsbeta]
12:39 <arturo> for the record, k8s etcd servers certificate changed (puppet based) and k8s just kept working [toolsbeta]
12:35 <arturo> according to `aborrero@cloud-cumin-01:~$ sudo cumin --force -x 'O{project:toolsbeta}' 'run-puppet-agent'` we are mostly back in business [toolsbeta]
12:14 <arturo> try switching all VMs to toolsbeta-puppetmaster-04 [toolsbeta]
12:14 <arturo> poweroff toolsbeta-puppetmaster-03 [toolsbeta]
12:12 <arturo> copy over labs/private from toolsbeta-puppetmaster-03 to toolsbeta-puppetmaster-04 [toolsbeta]
11:53 <arturo> create VM toolsbeta-puppetmaster-04 [toolsbeta]
11:35 <arturo> try reinstalling the python3 stack in toolsbeta-puppetmaster-03, because everything python-related segfaults [toolsbeta]
11:33 <arturo> reboot toolsbeta-puppetmaster-03 to try cleaning up potential kernel/filesystem problems [toolsbeta]
11:32 <arturo> apparently every python script segfaults in toolsbeta-puppetmaster-03 [toolsbeta]
11:27 <arturo> puppetdb wasn't the problem. The problem is puppet-enc segfaulting in toolsbeta-puppetmaster-03 [toolsbeta]
11:21 <arturo> puppet not working bc puppetdb, run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` [toolsbeta]
2020-06-04 §
21:06 <andrewbogott> added krenair to toolsbeta.admin group in ldap [toolsbeta]
2020-05-28 §
11:27 <arturo> cleanup livehackings [toolsbeta]
10:31 <arturo> livehacking puppetmaster and toolsbeta-proxy-1 to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 (T253816) [toolsbeta]
10:30 <arturo> livehacking puppetmaster to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 [toolsbeta]
2020-05-27 §
12:02 <arturo> the k8s cluster is now running v1.16.10 (T246122) [toolsbeta]
11:05 <arturo> trying `modules/kubeadm/files/wmcs-k8s-node-upgrade.py --control toolsbeta-test-k8s-control-1 --project toolsbeta --domain eqiad.wmflabs --src-version 1.15 --dst-version 1.16.10 -n toolsbeta-test-k8s-worker-1 -n toolsbeta-test-k8s-worker-2 -n toolsbeta-test-k8s-worker-3` (T246122) [toolsbeta]
11:02 <arturo> upgraded the rest of the k8s control plane nodes to 1.16.10 (T246122) [toolsbeta]
10:58 <arturo> running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo apt-get install kubelet -y` in the 1.16 version from the component repo (T246122) [toolsbeta]
10:58 <arturo> running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` and this time it works! (T246122) [toolsbeta]
2020-05-26 §
16:17 <bstorm_> fix incorrect volume name in kubeadm-config T246122 [toolsbeta]
15:02 <arturo> first k8s upgrade failed for yet-to-be-known reasons (T246122) [toolsbeta]
14:54 <arturo> `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` (T246122) [toolsbeta]
14:54 <arturo> bump installed version of kubeadm and kubectl to 1.16.10 (T246122) [toolsbeta]