I have several k3s clusters, one is misbehaving, wherein argocd was warning of not being able to contact github.com, upon investifation I found that all hostnames were being resolved to: 15.197.172.60
which resovles to some amazon eaccelerator:
dig -x 15.197.172.60 +short
a63452c77db78f54b.awsglobalaccelerator.com.
for example I try openssl on my nginx-ingress:
k exec -n ingress-nginx ingress-nginx-controller-56cc7c9475-skb8d -it -- openssl s_client -connect apple.com:443 -servername apple.com
\\Connecting to 15.197.172.60
CONNECTED(00000003)
289B45D0DF7F0000:error:0A000458:SSL routines:ssl3_read_bytes:tlsv1 unrecognized name:ssl/record/rec_layer_s3.c:916:SSL alert number 112
---
no peer certificate available
---
No client certificate CA names sent
Negotiated TLS1.3 group: <NULL>
---
SSL handshake has read 7 bytes and written 1542 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Protocol: TLSv1.3
This TLS version forbids renegotiation.
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
command terminated with exit code 1
DNS works fine from the kube-dns service:
kubectl port-forward -n kube-system svc/kube-dns 1053:53
Forwarding from 127.0.0.1:1053 -> 53
Forwarding from [::1]:1053 -> 53
Handling connection for 1053
Handling connection for 1053
dig @127.0.0.1 +tcp -p1053 apple.com +short
17.253.144.10
It was suggested that I elminate any wildcard entries in my dhcp server:
domain example.net
search example.org example.com
nameserver 10.1.2.1
forcing a very simple resolv.conf for k3s does seem to help for the initial node:
export K3S_RESOLV_CONF=/root/resolv.conf
cat /root/resolv.conf
nameserver 1.1.1.1
nameserver 8.8.8.8
However, if I start a multi-node cluster certain pods that spin up on the other joined nodes still have this issue, even if I have the above simple resov.conf and exported variables when joining the nodes. Can anyone explain? or offer a fix? What is going on with the 15.197.172.60 how are things being resolved to that address?