Kubernetes/Cluster

From Omnia
Jump to navigation Jump to search

Create Cluster

References:

How To Setup Kubernetes Cluster Using Kubeadm - Easy Guide
https://devopscube.com/setup-kubernetes-cluster-kubeadm/
Creating a cluster with kubeadm | Kubernetes
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

Install kubeadm

Reference:

Installing kubeadm | Kubernetes
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

Install dependencies:

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

Install kubelet kubeadm kubectl

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Setup Docker Runtime with Shim

Note: Technically Docker seems to have had a falling out with Kubernetes, but as it is still popular, you can still use it with cri-dockerd shim...

Install Docker:

apt install docker.io

Install shim: [1]

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.8/cri-dockerd-0.3.8.amd64.tgz
tar xvf cri-dockerd-0.3.8.amd64.tgz
sudo mv ./cri-dockerd /usr/local/bin/
ln -s /usr/local/bin/cri-dockerd /usr/bin/cri-dockerd

Initalize Cluster

Run pre-flight checks:

sudo kubeadm init phase preflight

Pre pull images:

sudo kubeadm config images pull

Fake initalize cluster:

# kubeadm init --dry-run
sudo kubeadm init --dry-run --cri-socket=unix:///var/run/cri-dockerd.sock

Initalize cluster:

# kubeadm init
sudo kubeadm init --cri-socket=unix:///var/run/cri-dockerd.sock --pod-network-cidr=10.244.0.0/16

Note: the "--pod-network-cidr=10.244.0.0/16" is needed for Flannel. Silly Flannel.

Initalized Success

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.x.x.x:6443 --token exxxxxxx.xxxxxxxxxxxx \
        --discovery-token-ca-cert-hash sha256:4xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Initalize Pod Network - Flannel

export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
kubectl get pods --all-namespaces

reference:

GitHub - flannel-io/flannel: flannel is a network fabric for containers, designed for Kubernetes
https://github.com/flannel-io/flannel#deploying-flannel-manually

Enable iptables Bridged Traffic on all the Nodes

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

Disable swap on all the Nodes

Kubernetes doesn't like Swap. Disable it.

root@kube1:~# kubeadm init
[init] Using Kubernetes version: v1.29.0
[preflight] Running pre-flight checks
        [WARNING Swap]: swap is supported for cgroup v2 only; the NodeSwap feature gate of the kubelet is beta but disabled by default

Disable swap:

sudo swapoff -a
(crontab -l 2>/dev/null; echo "@reboot /sbin/swapoff -a") | crontab - || true

Install Docker

ref [2]

Setup Repo

sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo \
 "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
 "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
 sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

ref [3]

Install Docker Enginer

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

ref [4]

Install docker and plugin

sudo apt-get update
sudo apt-get install docker-compose-plugin

ref [5]

Install Container Runtime

Install cri-dockerd binary:

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.8/cri-dockerd-0.3.8.amd64.tgz
tar xvf cri-dockerd-0.3.8.amd64.tgz
sudo mv cri-dockerd/cri-dockerd /usr/local/bin/
ln -s /usr/local/bin/cri-dockerd /usr/bin/cri-dockerd 

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.2.0/cri-dockerd-v0.2.0-linux-amd64.tar.gz
tar xvf cri-dockerd-v0.2.0-linux-amd64.tar.gz
sudo mv ./cri-dockerd /usr/local/bin/
ln -s /usr/local/bin/cri-dockerd /usr/bin/cri-dockerd 

cri-dockerd --help

Install service:

wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.service
wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.socket
sudo mv cri-docker.socket cri-docker.service /etc/systemd/system/
sudo sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
systemctl daemon-reload
systemctl enable cri-docker.service
systemctl enable --now cri-docker.socket
systemctl start cri-docker.service
systemctl start cri-docker.socket
systemctl status cri-docker.service
systemctl status cri-docker.socket
journalctl -u cri-docker.service
journalctl -u cri-docker.socket
crictl info
crictl --runtime-endpoint unix:///var/run/cri-dockerd.sock info
/etc/crictl.yaml
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: true
runtime-endpoint: unix:///var/run/cri-dockerd.sock
image-endpoint: unix:///var/run/cri-dockerd.sock
timeout: 10
debug: true
  1. /var/lib/kubelet/kubeadm-flags.env

ref [6]


Old:

git clone https://github.com/Mirantis/cri-dockerd.git

cd cri-dockerd
mkdir bin
VERSION=$((git describe --abbrev=0 --tags | sed -e 's/v//') || echo $(cat VERSION)-$(git log -1 --pretty='%h')) PRERELEASE=$(grep -q dev <<< "${VERSION}" && echo "pre" || echo "") REVISION=$(git log -1 --pretty='%h')
go build -ldflags="-X github.com/Mirantis/cri-dockerd/version.Version='$VERSION}' -X github.com/Mirantis/cri-dockerd/version.PreRelease='$PRERELEASE' -X github.com/Mirantis/cri-dockerd/version.BuildTime='$BUILD_DATE' -X github.com/Mirantis/cri-dockerd/version.GitCommit='$REVISION'" -o cri-dockerd

...

ref [7]

Install Kubeadm & Kubelet & Kubectl on all Nodes

sudo apt-get update && \
sudo apt-get install -y apt-transport-https ca-certificates curl && \
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
#echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys B53DC80D13EDEF05
sudo apt-get update -y && \
sudo apt-get install -y kubelet kubeadm kubectl && \
sudo apt-mark hold kubelet kubeadm kubectl


sudo apt-get update && \
sudo apt-get install -y kubelet=1.26.5-00 kubeadm=1.26.5-00 kubectl=1.26.5-00 docker.io && \
sudo apt-mark hold kubelet kubeadm kubectl docker.io
sudo mkdir /etc/containerd && \
sudo containerd config default > /etc/containerd/config.toml && \
sudo sed -i 's/            SystemdCgroup = false/            SystemdCgroup = true/' /etc/containerd/config.toml && \
sudo systemctl restart containerd && \
sudo systemctl restart kubelet

Initialize Kubeadm On Master Node To Setup Control Plane

sudo kubeadm init

Sockets found:

unix:///var/run/containerd/containerd.sock
unix:///var/run/cri-dockerd.sock
sudo kubeadm init --cri-socket=unix:///var/run/cri-dockerd.sock

Tear Down Cluster

If you need to try again:

kubeadm reset

Sort of, this doesn't do a great job...

Notes

How To Setup Kubernetes Cluster Using Kubeadm - Easy Guide
https://devopscube.com/setup-kubernetes-cluster-kubeadm/
How to Set Up Three Node Kubernetes Cluster on Ubuntu
https://k21academy.com/docker-kubernetes/three-node-kubernetes-cluster/
5 Fastest Ways to Install a k8s Cluster - Your DevOps Mentor
https://yourdevopsmentor.com/blog/5-ways-to-install-a-kubernetes-cluster/
Kubernetes Cluster setup on Ubuntu 22.04 using kubeadm with Calico, By Sir Babar Zahoor
https://www.linkedin.com/pulse/kubernetes-cluster-setup-ubuntu-2204-using-kubeadm-calico-md-sajjad
How to Set Up Three Node Kubernetes Cluster on Ubuntu 20.04
https://www.atlantic.net/dedicated-server-hosting/how-to-set-up-three-node-kubernetes-cluster-on-ubuntu-20-04/
Using Kubeadm to Create a Cluster | Airplane
https://www.airplane.dev/blog/kubeadm
Install Mirantis cri-dockerd as Docker Engine shim for Kubernetes | ComputingForGeeks
https://computingforgeeks.com/install-mirantis-cri-dockerd-as-docker-engine-shim-for-kubernetes/
Install Kubernetes Cluster on Ubuntu 22.04 with kubeadm | ComputingForGeeks
https://computingforgeeks.com/install-kubernetes-cluster-ubuntu-jammy/
Install Kubernetes 1.26 on Ubuntu 20.04 or 22.04 LTS | by Akriotis Kyriakos | Medium
https://akyriako.medium.com/install-kubernetes-on-ubuntu-20-04-f1791e8cf799
How to Install Kubernetes Cluster on Ubuntu 22.04
https://www.linuxtechi.com/install-kubernetes-on-ubuntu-22-04/

keywords