I write a ansible playbook for Kubernetes environment (timedata, swap, modules, containerd, kubeadm, kubelet, kubectl, vs). When I run the playbook, all operations in it are successful. But when I control the on vm, kubelet is not working
This is my playbook
- name: K8S Worker Deploybook
hosts:
- worker
- master
become: yes
gather_facts: no
vars_prompt:
- name: k8s_version
prompt: "Kubernetes version"
default: latest
private: no
tasks:
# Swap yapılandırması
- name: Sistem saat yapılandırması
shell:
cmd: hwclock --hctosys
- name: Swap alanlarının kapatılması
shell:
cmd: swapoff -a
- name: /etc/fstab içerisindeki swap tanımlarının önüne diyez ekler
shell:
cmd: sed -i 's/^(.*swap)/#1/g' /etc/fstab
# Tarih saat ayarları
- name: Install chrony download with apt
apt:
name: chrony
state: present
- name: Chrony configuration on debian based
blockinfile:
path: /etc/chrony/chrony.conf
block: |
server 0.tr.pool.ntp.org
server 1.tr.pool.ntp.org
server 2.tr.pool.ntp.org
server 3.tr.pool.ntp.org
state: present
- name: Service
service:
name: chronyd
state: restarted
enabled: yes
- name: Timezone configuration
timezone:
name: Europe/Istanbul
# Tüm paketlerin güncellenmesi
- name: Sistemdeki tüm paketlerin güncellenmesi
apt:
name: "*"
state: latest
update_cache: yes
# Modül tanımlarının yapılması
- name: overlay modülünün aktif edilmesi
shell:
cmd: modprobe overlay
- name: br_netfilter modülünün aktifleştirilmesi
shell:
cmd: modprobe br_netfilter
- name: Modüllerin konfigürasyon dosyasına işlenmesi
blockinfile:
path: /etc/modules-load.d/k8s.conf
block: |
overlay
br_netfilter
state: present
create: yes
- name: Network kurallarının tanımlanması
blockinfile:
path: /etc/sysctl.d/k8s.conf
block: |
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
state: present
create: yes
- name: Yapılan ayarların sisteme uygulanması
shell:
cmd: sysctl --system
# Containerd kurulumnları
- name: Containerd paket deposunun gpg keylerinin eklenmesi
shell:
cmd: curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker.gpg
- name: Containerd paket depolarının eklenmesi
shell:
cmd: echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu jammy stable" | sudo tee /etc/apt/sources.list.d/docker.list
- name: Paket depolarının güncellenmesi
apt:
update_cache: yes
- name: Containerd kurulması
apt:
name: containerd.io
state: latest
- name: Sistemdeki deamonların yeniden başlatılması
ansible.builtin.systemd:
daemon_reload: true
- name: Containerd servisini başlat ve yeniden başlatma sonrasında başlayacak şekilde konumlandır
ansible.builtin.systemd:
name: containerd
state: started
enabled: true
- name: Containerd konfigürasyon dizininin oluşturulması
file:
path: /etc/containerd
owner: root
group: root
mode: "0755"
state: directory
- name: Containerd default konfigürasyonun dosyaya yazılması
shell:
cmd: containerd config default | tee /etc/containerd/config.toml
- name:
shell:
cmd: sed -i 's/ SystemdCgroup = false/ SystemdCgroup = true/' /etc/containerd/config.toml
- name: Containerd servisinin yeniden başlatılması
service:
name: containerd
state: restarted
# Kubernetes bileşenlerinin kurulması
- name: APT Update
apt:
update_cache: yes
- name: Pre-required packages
apt:
pkg:
- apt-transport-https
- ca-certificates
- curl
- name: Kubernets paket deposunun gpg keylerinin eklenmesi
shell:
cmd: curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg
- name: Kubernetes paket depolarının eklenmesi
shell:
cmd: echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
- name: APT Update
apt:
update_cache: yes
- name: Install kubelet
apt:
name: kubelet
state: "{{k8s_version}}"
- name: Install kubeadm
apt:
name: kubeadm
state: "{{k8s_version}}"
- name: Install kubectl
apt:
name: kubectl
state: "{{k8s_version}}"
- name: Hold kubelet
dpkg_selections:
name: kubelet
selection: hold
- name: Hold kubeadm
dpkg_selections:
name: kubeadm
selection: hold
- name: Hold kubectl
dpkg_selections:
name: kubectl
selection: hold
- name: Remove dependencies that are no longer required
apt:
autoremove: yes
- name: Remove useless packages from the cache
apt:
autoclean: yes
I see this error on journalctl
Oct 22 20:12:58 kube01 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.780055 62503 server.go:467] "Kubelet version" kubeletVersion="v1.28.2"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.780142 62503 server.go:469] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.780560 62503 server.go:630] "Standalone mode, no API client"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.794365 62503 server.go:518] "No api server defined - no events will be sent to API server"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.794385 62503 server.go:725] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.794549 62503 container_manager_linux.go:265] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.794703 62503 container_manager_linux.go:270] "Creating Container Manager object based on Node Config" nodeConfig={"RuntimeCgroupsName":"","SystemCgroupsName":"","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"cgroupfs","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":false,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":null,"HardEvictionThresholds":[],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"ExperimentalMemoryManagerPolicy":"None","ExperimentalMemoryManagerReservedMemory":null,"PodPidsLimit":-1,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null}
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.794723 62503 topology_manager.go:138] "Creating topology manager with none policy"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.794731 62503 container_manager_linux.go:301] "Creating device plugin manager"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.794800 62503 state_mem.go:36] "Initialized new in-memory state store"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.794840 62503 kubelet.go:399] "Kubelet is running in standalone mode, will skip API server sync"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.795167 62503 kuberuntime_manager.go:257] "Container runtime initialized" containerRuntime="containerd" version="1.6.24" apiVersion="v1"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.795437 62503 volume_host.go:74] "KubeClient is nil. Skip initialization of CSIDriverLister"
Oct 22 20:12:58 kube01 kubelet[62503]: W1022 20:12:58.795559 62503 probe.go:268] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
Oct 22 20:12:58 kube01 kubelet[62503]: W1022 20:12:58.795885 62503 csi_plugin.go:189] kubernetes.io/csi: kubeclient not set, assuming standalone kubelet
Oct 22 20:12:58 kube01 kubelet[62503]: W1022 20:12:58.795998 62503 csi_plugin.go:266] Skipping CSINode initialization, kubelet running in standalone mode
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.796279 62503 server.go:1232] "Started kubelet"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.796422 62503 server.go:162] "Starting to listen" address="0.0.0.0" port=10250
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.796435 62503 kubelet.go:1579] "No API server defined - no node status update will be sent"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.796455 62503 server.go:194] "Starting to listen read-only" address="0.0.0.0" port=10255
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.797766 62503 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.796499 62503 ratelimit.go:65] "Setting rate limiting for podresources endpoint" qps=100 burstTokens=10
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.798430 62503 server.go:233] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 22 20:12:58 kube01 kubelet[62503]: E1022 20:12:58.796812 62503 cri_stats_provider.go:448] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs"
Oct 22 20:12:58 kube01 kubelet[62503]: E1022 20:12:58.798457 62503 kubelet.go:1431] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.798432 62503 server.go:462] "Adding debug handlers to kubelet server"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.799218 62503 volume_manager.go:291] "Starting Kubelet Volume Manager"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.799280 62503 desired_state_of_world_populator.go:151] "Desired state populator starts to run"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.799325 62503 reconciler_new.go:29] "Reconciler: start to sync state"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.821444 62503 cpu_manager.go:214] "Starting CPU manager" policy="none"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.821460 62503 cpu_manager.go:215] "Reconciling" reconcilePeriod="10s"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.821471 62503 state_mem.go:36] "Initialized new in-memory state store"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.847618 62503 policy_none.go:49] "None policy: Start"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.848345 62503 memory_manager.go:169] "Starting memorymanager" policy="None"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.848373 62503 state_mem.go:35] "Initializing new in-memory state store"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.872491 62503 manager.go:471] "Failed to read data from checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.872776 62503 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.899854 62503 desired_state_of_world_populator.go:159] "Finished populating initial desired state of world"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.910726 62503 kubelet_network_linux.go:50] "Initialized iptables rules." protocol="IPv4"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.913220 62503 kubelet_network_linux.go:50] "Initialized iptables rules." protocol="IPv6"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.913254 62503 status_manager.go:213] "Kubernetes client is nil, not starting status manager"
Oct 22 20:12:58 kube01 kubelet[62503]: I1022 20:12:58.913264 62503 kubelet.go:2303] "Starting kubelet main sync loop"
Oct 22 20:12:58 kube01 kubelet[62503]: E1022 20:12:58.913299 62503 kubelet.go:2327] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 22 20:13:08 kube01 systemd[1]: kubelet.service: Current command vanished from the unit file, execution of the command list won't be resumed.
Oct 22 20:13:08 kube01 systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
Oct 22 20:13:08 kube01 systemd[1]: kubelet.service: Deactivated successfully.
Oct 22 20:13:08 kube01 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Oct 22 20:13:08 kube01 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Oct 22 20:13:08 kube01 kubelet[62751]: E1022 20:13:08.519080 62751 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory"
Oct 22 20:13:08 kube01 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Oct 22 20:13:08 kube01 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Oct 22 20:13:18 kube01 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1.
Oct 22 20:13:18 kube01 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Oct 22 20:13:18 kube01 systemd[1]: Started kubelet: The Kubernetes Node Agent.
Oct 22 20:13:18 kube01 kubelet[62995]: E1022 20:13:18.805505 62995 run.go:74] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory"
Oct 22 20:13:18 kube01 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Oct 22 20:13:18 kube01 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Oct 22 20:13:28 kube01 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 2.
Oct 22 20:13:28 kube01 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
I check /var/lib/kubelet/config.yaml file but there is no. When I write a line by line in playbook it’s worked but when playbook installed, kubelet not working.
Why might this be happening?
Do you any suggest?
