Run multiple minikube Kubernetes clusters on Ubuntu Linux with KVM

Ramp up your Kubernetes development, CI-tooling or testing workflow by running multiple Kubernetes clusters on Ubuntu Linux with KVM and minikube.

Pre-reqs

Use Ubuntu 16.04 as a base installation.
You'll need nested virtualization enabled or a bare metal machine in the cloud. You can find these at Packet.net or Scaleway.

Install Tooling

Install KVM

KVM enables virtualization on Linux, but other options are also available for use with minikube such as: virtualbox/vmwarefusion/kvm/xhyve/hyperv.

Follow instructions here to install packages from apt.

Install kubectl

https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x kubectl
sudo mv kubectl /usr/local/bin/

Install minikube and the KVM2 driver

curl -SLO https://github.com/kubernetes/minikube/releases/download/v0.28.2/docker-machine-driver-kvm2
curl -SLO https://github.com/kubernetes/minikube/releases/download/v0.28.2/minikube-linux-amd64

chmod +x docker-machine-driver-kvm2
chmod +x minikube-linux-amd64

sudo mv docker-machine-driver-kvm2 /usr/local/bin
sudo mv minikube-linux-amd64 /usr/local/bin/minikube

Create a cluster with `kubeadm`

Using the kubeadm bootstrapper will enable RBAC

Create your first cluster VM:

minikube start --bootstrapper=kubeadm --vm-driver=kvm2 --memory 4096 --cpus 4 --profile cluster1

You can set up additional separate VMs using the --profile flag.

Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Downloading Minikube ISO
 160.27 MB / 160.27 MB [============================================] 100.00% 0s
Getting VM IP address...
Moving files into cluster...
Downloading kubeadm v1.10.0
Downloading kubelet v1.10.0
Finished Downloading kubelet v1.10.0
Finished Downloading kubeadm v1.10.0
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from config file.

You can manage each cluster using kubectl by switching between contexts saved in ~/.kube/config.yaml. The kubectx tool is also popular in the community for switching between these quickly.

kubectl config get-contexts
CURRENT   NAME       CLUSTER    AUTHINFO   NAMESPACE
*         cluster1   cluster1   cluster1

Working with the various IPs

Pass the --profile flag to minikube commands so that you can access the ports you want.

minikube ip --profile cluster1
192.168.39.10

You may want to use SSH port forwarding with ssh -L port:port [email protected] or kubectl port-forward to give access to services and NodePorts available on the minikube VM. If you're comfortable with iptables you could also set up some NAT rules, but I would recommend against it since the IP addresses of your minikube environments may change when being restarted.

Deploy something to test it out (OpenFaaS)

You can now install something to test the cluster.

Configure helm and tiller:

curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash
kubectl -n kube-system create sa tiller \
  && kubectl create clusterrolebinding tiller   \
  --clusterrole cluster-admin   \
  --serviceaccount=kube-system:tiller
helm init --skip-refresh --upgrade --service-account tiller

Setup OpenFaaS via helm:

kubectl apply -f https://raw.githubusercontent.com/openfaas/faas-netes/master/namespaces.yml

helm repo add openfaas https://openfaas.github.io/faas-netes/

helm repo update  && helm upgrade openfaas \
  --install openfaas/openfaas  \
  --namespace openfaas  \
  --set functionNamespace=openfaas-fn

Now access the OpenFaaS gateway via the NodePort:

curl http://$(minikube ip --profile cluster1):31112/system/info

{"provider":{"provider":"faas-netes","version":{"sha":"5539cf43c15a28e9af998cdc25b5da06252b62e1","release":"0.6.0"},"orchestration":"kubernetes"},"version":{"commit_message":"Attach X-Call-Id to asynchronous calls","sha":"c86de503c7a20a46645239b9b081e029b15bf69b","release":"0.8.11"}}

Port forward from your laptop to the Linux machine:

Find the IP of the minikube VM with echo $(minikube ip --profile cluster1) i.e. 192.168.39.10

ssh -L 31112:192.168.39.10:31112 user@linux-host

You can now access the OpenFaaS installation from your remote Linux host via http://127.0.0.1:31112 or even using faas-cli --gateway 127.0.0.1:31112.

You can also open up your OpenFaaS installation to your friends or for testing public webhooks via ngrok.

./ngrok http $(minikube ip --profile cluster1):31112

Wrapping up

If you have comments, questions or suggestions feel free to reach out over Twitter @alexellisuk

joglomedia/kvm_minikube.md