Kubernetes/Cluster
Jump to navigation
Jump to search
Subpage Table of Contents
Create Cluster
References:
How To Setup Kubernetes Cluster Using Kubeadm - Easy Guide https://devopscube.com/setup-kubernetes-cluster-kubeadm/
Creating a cluster with kubeadm | Kubernetes https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
Install kubeadm
Reference:
Installing kubeadm | Kubernetes https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
Install dependencies:
sudo apt-get update sudo apt-get install -y apt-transport-https ca-certificates curl gpg curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
Install kubelet kubeadm kubectl
sudo apt-get update sudo apt-get install -y kubelet kubeadm kubectl sudo apt-mark hold kubelet kubeadm kubectl
Install Container Runtime
Install Containerd
apt install containerd
mkdir /etc/containerd containerd config default | tee /etc/containerd/config.toml > /dev/null
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
systemctl daemon-reload systemctl start containerd
fix ref: [1]
Note to view the images and containers:
crictl images crictl ps -a
sudo ctr ns ls
sudo ctr -n k8s.io images ls sudo ctr -n k8s.io containers ls # or 'c ls'
OR Setup Docker Runtime with Shim
Note: Technically Docker seems to have had a falling out with Kubernetes, but as it is still popular, you can still use it with cri-dockerd shim...
Install Docker:
apt install docker.io
See Kubernetes/Cluster#Install_Container_Runtime
Disable swap on all the Nodes
WARNING: Kubernetes doesn't like Swap. Disable it. 'kubeadm init' only says warning, but kubelet will fail to start if swap is installed!
Only warning here, but really it should be fatal!
root@kube1:~# kubeadm init
[init] Using Kubernetes version: v1.29.0
[preflight] Running pre-flight checks
[WARNING Swap]: swap is supported for cgroup v2 only; the NodeSwap feature gate of the kubelet is beta but disabled by default
List swap:
swapon -s
Disable swap:
sudo swapoff -a (crontab -l 2>/dev/null; echo "@reboot /sbin/swapoff -a") | crontab - || true
Enable iptables Bridged Traffic on all the Nodes
Note: installing docker.io seems to do most of these for you (so if you go that route, you can skip these manual steps)...
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf overlay br_netfilter EOF sudo modprobe overlay ; sudo modprobe br_netfilter # sysctl params required by setup, params persist across reboots cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 EOF # Apply sysctl params without reboot sudo sysctl --system
Initalize Cluster
Run pre-flight checks:
sudo kubeadm init phase preflight
Pre pull images:
sudo kubeadm config images pull
Fake initalize cluster:
sudo kubeadm init --dry-run --pod-network-cidr=10.244.0.0/16
Note: the "--pod-network-cidr=10.244.0.0/16" is needed for Flannel. Silly Flannel.
Initalize cluster:
sudo kubeadm init --pod-network-cidr=10.244.0.0/16
Initalize with Docker
Run pre-flight checks:
sudo kubeadm init phase preflight --cri-socket=unix:///var/run/cri-dockerd.sock
Pre pull images:
sudo kubeadm config images pull --cri-socket=unix:///var/run/cri-dockerd.sock
Fake initalize cluster with Docker:
sudo kubeadm init --dry-run --pod-network-cidr=10.244.0.0/16 --cri-socket=unix:///var/run/cri-dockerd.sock
Note: the "--pod-network-cidr=10.244.0.0/16" is needed for Flannel. Silly Flannel.
Or Initalize cluster with Docker:
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --cri-socket=unix:///var/run/cri-dockerd.sock
Initalize with Redundant Control Plane
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=192.x.x.x:6443 --upload-certs --cri-socket=unix:///var/run/cri-dockerd.sock
With keepalived on same host:
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint=192.x.x.x:26443 --upload-certs --cri-socket=unix:///var/run/cri-dockerd.sock
Initalized Success
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.x.x.x:6443 --token exxxxxxx.xxxxxxxxxxxx \
--discovery-token-ca-cert-hash sha256:4xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
WARNING WARNING WARNING - before joining another node, make sure to add your pod network!
Get Join Command
kubeadm init phase upload-certs --upload-certs
kubeadm token create --print-join-command
Example:
kubeadm join 192.x.x.x:6443 --token xxx.xxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Example with Docker:
kubeadm join 192.x.x.x:6443 --token xxx.xxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx --cri-socket=unix:///var/run/cri-dockerd.sock
Initalize Pod Network - Flannel
export KUBECONFIG=/etc/kubernetes/admin.conf
k get pods -o wide --all-namespaces
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
kubectl get pods --all-namespaces # or just kubectl get pods -n kube-flannel
Note, if you need to retry, you can delete, then retry:
kubectl delete -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
Note, flannel expects the CIDR to be 10.244.0.0/16. If you have it set to something else, you will need to modify the manifest. To check what your cluster is configured for, or if you even specified the the cidr during cluster creation (required for Flannel, if it returns nothing that is a problem): [1]
kubectl cluster-info dump | grep cluster-cidr
reference:
GitHub - flannel-io/flannel: flannel is a network fabric for containers, designed for Kubernetes https://github.com/flannel-io/flannel#deploying-flannel-manually
Join Worker Nodes
1. Disable Swap
2. Configure
kubeadm join 192.x.x.x:6443 --token exxxxxxx.xxxxxxxxxxxx \
--discovery-token-ca-cert-hash sha256:4xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Test a few pods
kubectl run shell1 --image ubuntu sleep infinity kubectl run shell2 --image ubuntu sleep infinity kubectl run shell3 --image ubuntu sleep infinity kubectl run shell4 --image ubuntu sleep infinity kubectl run shell5 --image ubuntu sleep infinity
Delete Node
How to delete a node from Kubernetes [2]
kubectl drain <node-name>
Might have to ignore some deamonsets:
kubectl drain <node-name> --ignore-daemonsets --delete-local-data
Delete node:
kubectl delete node <node-name>
Reset node locally: (run locally)
kubeadm reset
Install Docker
Overview of installing Docker Compose | Docker Docs https://docs.docker.com/compose/install/
Setup Repo
sudo apt-get update sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo \ "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
ref [3]
Install Docker Enginer
sudo apt-get update sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
ref [2]
Install docker and plugin
sudo apt-get update sudo apt-get install docker-compose-plugin
ref [3]
Install Container Runtime
Install cri-dockerd binary:
wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.8/cri-dockerd-0.3.8.amd64.tgz tar xvf cri-dockerd-0.3.8.amd64.tgz sudo mv cri-dockerd/cri-dockerd /usr/local/bin/ rmdir cri-dockerd/ ln -s /usr/local/bin/cri-dockerd /usr/bin/cri-dockerd
wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.2.0/cri-dockerd-v0.2.0-linux-amd64.tar.gz tar xvf cri-dockerd-v0.2.0-linux-amd64.tar.gz sudo mv ./cri-dockerd /usr/local/bin/ ln -s /usr/local/bin/cri-dockerd /usr/bin/cri-dockerd
cri-dockerd --help
Install service:
wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.service wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.socket sudo mv cri-docker.socket cri-docker.service /etc/systemd/system/ sudo sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
systemctl daemon-reload systemctl enable cri-docker.service systemctl enable --now cri-docker.socket
systemctl start cri-docker.service systemctl start cri-docker.socket
systemctl status cri-docker.service systemctl status cri-docker.socket
journalctl -u cri-docker.service journalctl -u cri-docker.socket
crictl info crictl --runtime-endpoint unix:///var/run/cri-dockerd.sock info
/etc/crictl.yaml
runtime-endpoint: unix:///var/run/containerd/containerd.sock image-endpoint: unix:///var/run/containerd/containerd.sock timeout: 10 debug: true
runtime-endpoint: unix:///var/run/cri-dockerd.sock image-endpoint: unix:///var/run/cri-dockerd.sock timeout: 10 debug: true
- /var/lib/kubelet/kubeadm-flags.env
ref [4]
Old:
git clone https://github.com/Mirantis/cri-dockerd.git
cd cri-dockerd
mkdir bin
VERSION=$((git describe --abbrev=0 --tags | sed -e 's/v//') || echo $(cat VERSION)-$(git log -1 --pretty='%h')) PRERELEASE=$(grep -q dev <<< "${VERSION}" && echo "pre" || echo "") REVISION=$(git log -1 --pretty='%h')
go build -ldflags="-X github.com/Mirantis/cri-dockerd/version.Version='$VERSION}' -X github.com/Mirantis/cri-dockerd/version.PreRelease='$PRERELEASE' -X github.com/Mirantis/cri-dockerd/version.BuildTime='$BUILD_DATE' -X github.com/Mirantis/cri-dockerd/version.GitCommit='$REVISION'" -o cri-dockerd
...
ref [5]
Install Kubeadm & Kubelet & Kubectl on all Nodes
sudo apt-get update && \ sudo apt-get install -y apt-transport-https ca-certificates curl && \ sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
#echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys B53DC80D13EDEF05
sudo apt-get update -y && \ sudo apt-get install -y kubelet kubeadm kubectl && \ sudo apt-mark hold kubelet kubeadm kubectl
sudo apt-get update && \ sudo apt-get install -y kubelet=1.26.5-00 kubeadm=1.26.5-00 kubectl=1.26.5-00 docker.io && \ sudo apt-mark hold kubelet kubeadm kubectl docker.io
sudo mkdir /etc/containerd && \ sudo containerd config default > /etc/containerd/config.toml && \ sudo sed -i 's/ SystemdCgroup = false/ SystemdCgroup = true/' /etc/containerd/config.toml && \ sudo systemctl restart containerd && \ sudo systemctl restart kubelet
Initialize Kubeadm On Master Node To Setup Control Plane
sudo kubeadm init
Sockets found:
unix:///var/run/containerd/containerd.sock unix:///var/run/cri-dockerd.sock
sudo kubeadm init --cri-socket=unix:///var/run/cri-dockerd.sock
Tear Down
If you need to reset a node (even controller) locally:
kubeadm reset
If with docker:
kubeadm reset --cri-socket=unix:///var/run/cri-dockerd.sock
Useful Debug
crictl images crictl ps -a sudo ctr ns ls sudo ctr -n k8s.io images ls sudo ctr -n k8s.io containers ls # or 'c ls'
service kubelet status journalctl -u kubelet
Notes
How To Setup Kubernetes Cluster Using Kubeadm - Easy Guide https://devopscube.com/setup-kubernetes-cluster-kubeadm/
How to Set Up Three Node Kubernetes Cluster on Ubuntu https://k21academy.com/docker-kubernetes/three-node-kubernetes-cluster/
5 Fastest Ways to Install a k8s Cluster - Your DevOps Mentor https://yourdevopsmentor.com/blog/5-ways-to-install-a-kubernetes-cluster/
Kubernetes Cluster setup on Ubuntu 22.04 using kubeadm with Calico, By Sir Babar Zahoor https://www.linkedin.com/pulse/kubernetes-cluster-setup-ubuntu-2204-using-kubeadm-calico-md-sajjad
How to Set Up Three Node Kubernetes Cluster on Ubuntu 20.04 https://www.atlantic.net/dedicated-server-hosting/how-to-set-up-three-node-kubernetes-cluster-on-ubuntu-20-04/
Using Kubeadm to Create a Cluster | Airplane https://www.airplane.dev/blog/kubeadm
Install Mirantis cri-dockerd as Docker Engine shim for Kubernetes | ComputingForGeeks https://computingforgeeks.com/install-mirantis-cri-dockerd-as-docker-engine-shim-for-kubernetes/
Install Kubernetes Cluster on Ubuntu 22.04 with kubeadm | ComputingForGeeks https://computingforgeeks.com/install-kubernetes-cluster-ubuntu-jammy/
Install Kubernetes 1.26 on Ubuntu 20.04 or 22.04 LTS | by Akriotis Kyriakos | Medium https://akyriako.medium.com/install-kubernetes-on-ubuntu-20-04-f1791e8cf799
How to Install Kubernetes Cluster on Ubuntu 22.04 https://www.linuxtechi.com/install-kubernetes-on-ubuntu-22-04/
keywords
References
- ↑ https://devops.stackexchange.com/questions/5898/how-to-get-kubernetes-pod-network-cidr
- ↑ How to gracefully remove a node from Kubernetes? - Stack Overflow - https://stackoverflow.com/questions/35757620/how-to-gracefully-remove-a-node-from-kubernetes
- ↑ Install Docker Engine on Ubuntu - https://docs.docker.com/engine/install/ubuntu/#set-up-the-repository