2020-09-09
§
|
11:50 |
<arturo> |
after force-rebooting everything, the k8s cluster seems to have recovered itself. magic. |
[toolsbeta] |
11:45 |
<arturo> |
force-rebooting the 3 k8s etcd nodes. They seem down |
[toolsbeta] |
11:42 |
<arturo> |
actually, the whole k8s cluster seems down? the API seems down at least |
[toolsbeta] |
11:39 |
<arturo> |
all 3 k8s control nodes seem in bad shape. Wont let me ssh in, or use the console access. Try force-rebooting them |
[toolsbeta] |
11:27 |
<arturo> |
created 2 VMs: toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2 (T250172) |
[toolsbeta] |
11:25 |
<arturo> |
created new server group toolsbeta-k8s-ingress (T250172) |
[toolsbeta] |
11:24 |
<arturo> |
created new puppet prefix `toolsbeta-test-k8s-ingress` (T250172) |
[toolsbeta] |
2020-06-11
§
|
12:42 |
<arturo> |
introduce puppet profile 'toolsbeta-docker-registry' and relocate some hiera config there |
[toolsbeta] |
12:39 |
<arturo> |
for the record, k8s etcd servers certificate changed (puppet based) and k8s just kept working |
[toolsbeta] |
12:35 |
<arturo> |
according to `aborrero@cloud-cumin-01:~$ sudo cumin --force -x 'O{project:toolsbeta}' 'run-puppet-agent'` we are mostly back in business |
[toolsbeta] |
12:14 |
<arturo> |
try switching all VMs to toolsbeta-puppetmaster-04 |
[toolsbeta] |
12:14 |
<arturo> |
poweroff toolsbeta-puppetmaster-03 |
[toolsbeta] |
12:12 |
<arturo> |
copy over labs/private from toolsbeta-puppetmaster-03 to toolsbeta-puppetmaster-04 |
[toolsbeta] |
11:53 |
<arturo> |
create VM toolsbeta-puppetmaster-04 |
[toolsbeta] |
11:35 |
<arturo> |
try reinstalling the python3 stack in toolsbeta-puppetmaster-03, because everything python-related segfaults |
[toolsbeta] |
11:33 |
<arturo> |
reboot toolsbeta-puppetmaster-03 to try cleaning up potential kernel/filesystem problems |
[toolsbeta] |
11:32 |
<arturo> |
apparently every python script segfaults in toolsbeta-puppetmaster-03 |
[toolsbeta] |
11:27 |
<arturo> |
puppetdb wasn't the problem. The problem is puppet-enc segfaulting in toolsbeta-puppetmaster-03 |
[toolsbeta] |
11:21 |
<arturo> |
puppet not working bc puppetdb, run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` |
[toolsbeta] |