Skip to content

Instantly share code, notes, and snippets.

@joglomedia
Forked from alexellis/kvm_minikube.md
Created August 28, 2021 14:57
Show Gist options
  • Save joglomedia/5bfbed3861a594621e41c015394f130e to your computer and use it in GitHub Desktop.
Save joglomedia/5bfbed3861a594621e41c015394f130e to your computer and use it in GitHub Desktop.

Revisions

  1. @alexellis alexellis revised this gist Sep 1, 2018. 1 changed file with 7 additions and 1 deletion.
    8 changes: 7 additions & 1 deletion kvm_minikube.md
    Original file line number Diff line number Diff line change
    @@ -17,7 +17,13 @@ Run all of these commands on your Linux host unless otherwise specified.

    KVM enables virtualization on Linux, but other options are also available for use with minikube such as: virtualbox/vmwarefusion/kvm/xhyve/hyperv.

    Follow [instructions here](https://blog.alexellis.io/kvm-kubernetes-primer/) to install packages from `apt`.
    Follow [instructions here](https://blog.alexellis.io/kvm-kubernetes-primer/) to install packages from `apt`:

    ```
    sudo apt-get install -qy \
    qemu-kvm libvirt-bin virtinst bridge-utils cpu-checker
    sudo kvm-ok
    ```

    ### Install kubectl

  2. @alexellis alexellis revised this gist Aug 22, 2018. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion kvm_minikube.md
    Original file line number Diff line number Diff line change
    @@ -6,7 +6,7 @@ We'll set up all the tooling so that you can build one or many single-node Kuber

    ## Pre-reqs

    * This tutorial uses Ubuntu 16.04 as a base installation, but other distributions are supported by KVM. You'll need to find out how to install KVM with your package manager.
    * This tutorial uses Ubuntu 16.04 as a base installation, but other distributions are supported by KVM. You'll need to find out how to install KVM with your package manager. If you're using [Fedora you can follow these instructions to Install KVM](https://docs.fedoraproject.org/en-US/quick-docs/getting-started-with-virtualization/).
    * You'll need nested virtualization available on a cloud host, a spare machine under your desk or a bare metal machine in the cloud. You can find affordable bare metal at [Scaleway](https://www.scaleway.com) or high-spec/performance bare-metal over at: [Packet.net](https://www.packet.net)

    ## Install Tooling
  3. @alexellis alexellis revised this gist Aug 22, 2018. 1 changed file with 8 additions and 0 deletions.
    8 changes: 8 additions & 0 deletions kvm_minikube.md
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,9 @@
    Ramp up your Kubernetes development, CI-tooling or testing workflow by running multiple Kubernetes clusters on Ubuntu Linux with KVM and minikube.

    In this tutorial we will combine the popular `minikube` tool with Linux's Kernel-based Virtual Machine (KVM) support. It is a great way to re-purpose an old machine that you found on eBay or have gathering gust under your desk. An Intel NUC would also make a great host for this tutorial if you want to buy some new hardware. Another popular angle is to use a bare metal host in the cloud and I've provided some details on that below.

    We'll set up all the tooling so that you can build one or many single-node Kubernetes clusters and then deploy applications to them such as OpenFaaS using familiar tooling like helm. I'll then show you how to access the Kubernetes clusters from a remote machine such as your laptop.

    ## Pre-reqs

    * This tutorial uses Ubuntu 16.04 as a base installation, but other distributions are supported by KVM. You'll need to find out how to install KVM with your package manager.
    @@ -146,4 +150,8 @@ You can also open up your OpenFaaS installation to your friends or for testing p

    ### Wrapping up

    We've now built one or many Kubernetes clusters using `minikube` and KVM using the `--profile` flag to separate them and assign each its own name. Port-forwarding, ngrok or SSH provided us with temporary access into the clusters for testing purposes. Where could you take this text?

    After you have gained some muscle-memory with creating and accessing these development clusters, you could go on to bake them into projects. A single-use cluster would be great to use in a CI/CD pipeline for end-to-end testing or any other tasks that require multiple configurations such as ensuring backwards compatibility with different Kubernetes versions or with RBAC enabled or disabled.

    If you have comments, questions or suggestions feel free to reach out over Twitter [@alexellisuk](https://twitter.com/alexellisuk)
  4. @alexellis alexellis revised this gist Aug 22, 2018. 1 changed file with 8 additions and 2 deletions.
    10 changes: 8 additions & 2 deletions kvm_minikube.md
    Original file line number Diff line number Diff line change
    @@ -3,10 +3,12 @@ Ramp up your Kubernetes development, CI-tooling or testing workflow by running m
    ## Pre-reqs

    * This tutorial uses Ubuntu 16.04 as a base installation, but other distributions are supported by KVM. You'll need to find out how to install KVM with your package manager.
    * You'll need nested virtualization enabled or a bare metal machine in the cloud. You can find these at [Packet.net](https://www.packet.net) or [Scaleway](https://www.scaleway.com).
    * You'll need nested virtualization available on a cloud host, a spare machine under your desk or a bare metal machine in the cloud. You can find affordable bare metal at [Scaleway](https://www.scaleway.com) or high-spec/performance bare-metal over at: [Packet.net](https://www.packet.net)

    ## Install Tooling

    Run all of these commands on your Linux host unless otherwise specified.

    ### Install KVM

    KVM enables virtualization on Linux, but other options are also available for use with minikube such as: virtualbox/vmwarefusion/kvm/xhyve/hyperv.
    @@ -122,6 +124,10 @@ curl http://$(minikube ip --profile cluster1):31112/system/info
    {"provider":{"provider":"faas-netes","version":{"sha":"5539cf43c15a28e9af998cdc25b5da06252b62e1","release":"0.6.0"},"orchestration":"kubernetes"},"version":{"commit_message":"Attach X-Call-Id to asynchronous calls","sha":"c86de503c7a20a46645239b9b081e029b15bf69b","release":"0.8.11"}}
    ```

    ### Access the cluster from your laptop

    You can also gain access into the cluster(s) from a remote machine.

    Port forward from your laptop to the Linux machine:

    Find the IP of the minikube VM with `echo $(minikube ip --profile cluster1)` i.e. `192.168.39.10`
    @@ -132,7 +138,7 @@ ssh -L -N 31112:192.168.39.10:31112 user@linux-host

    You can now access the OpenFaaS installation from your remote Linux host via http://127.0.0.1:31112 or even using `faas-cli --gateway 127.0.0.1:31112`.

    You can also open up your OpenFaaS installation to your friends or for testing public webhooks via [ngrok](https://ngrok.com/download).
    You can also open up your OpenFaaS installation to your friends or for testing public webhooks via [ngrok](https://ngrok.com/download). Run the tool on your Linux host

    ```
    ./ngrok http $(minikube ip --profile cluster1):31112
  5. @alexellis alexellis revised this gist Aug 22, 2018. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion kvm_minikube.md
    Original file line number Diff line number Diff line change
    @@ -127,7 +127,7 @@ Port forward from your laptop to the Linux machine:
    Find the IP of the minikube VM with `echo $(minikube ip --profile cluster1)` i.e. `192.168.39.10`

    ```
    ssh -L 31112:192.168.39.10:31112 user@linux-host
    ssh -L -N 31112:192.168.39.10:31112 user@linux-host
    ```

    You can now access the OpenFaaS installation from your remote Linux host via http://127.0.0.1:31112 or even using `faas-cli --gateway 127.0.0.1:31112`.
  6. @alexellis alexellis revised this gist Aug 21, 2018. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion kvm_minikube.md
    Original file line number Diff line number Diff line change
    @@ -2,7 +2,7 @@ Ramp up your Kubernetes development, CI-tooling or testing workflow by running m

    ## Pre-reqs

    * Use Ubuntu 16.04 as a base installation.
    * This tutorial uses Ubuntu 16.04 as a base installation, but other distributions are supported by KVM. You'll need to find out how to install KVM with your package manager.
    * You'll need nested virtualization enabled or a bare metal machine in the cloud. You can find these at [Packet.net](https://www.packet.net) or [Scaleway](https://www.scaleway.com).

    ## Install Tooling
  7. @alexellis alexellis revised this gist Aug 21, 2018. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion kvm_minikube.md
    Original file line number Diff line number Diff line change
    @@ -3,7 +3,7 @@ Ramp up your Kubernetes development, CI-tooling or testing workflow by running m
    ## Pre-reqs

    * Use Ubuntu 16.04 as a base installation.
    * You'll need nested virtualization enabled or a bare metal machine in the cloud. You can find these at Packet.net or Scaleway.
    * You'll need nested virtualization enabled or a bare metal machine in the cloud. You can find these at [Packet.net](https://www.packet.net) or [Scaleway](https://www.scaleway.com).

    ## Install Tooling

  8. @alexellis alexellis revised this gist Aug 21, 2018. 1 changed file with 6 additions and 0 deletions.
    6 changes: 6 additions & 0 deletions kvm_minikube.md
    Original file line number Diff line number Diff line change
    @@ -132,6 +132,12 @@ ssh -L 31112:192.168.39.10:31112 user@linux-host

    You can now access the OpenFaaS installation from your remote Linux host via http://127.0.0.1:31112 or even using `faas-cli --gateway 127.0.0.1:31112`.

    You can also open up your OpenFaaS installation to your friends or for testing public webhooks via [ngrok](https://ngrok.com/download).

    ```
    ./ngrok http $(minikube ip --profile cluster1):31112
    ```

    ### Wrapping up

    If you have comments, questions or suggestions feel free to reach out over Twitter [@alexellisuk](https://twitter.com/alexellisuk)
  9. @alexellis alexellis revised this gist Aug 21, 2018. 1 changed file with 8 additions and 2 deletions.
    10 changes: 8 additions & 2 deletions kvm_minikube.md
    Original file line number Diff line number Diff line change
    @@ -1,3 +1,5 @@
    Ramp up your Kubernetes development, CI-tooling or testing workflow by running multiple Kubernetes clusters on Ubuntu Linux with KVM and minikube.

    ## Pre-reqs

    * Use Ubuntu 16.04 as a base installation.
    @@ -7,7 +9,7 @@

    ### Install KVM

    https://www.linux-kvm.org/page/Main_Page
    KVM enables virtualization on Linux, but other options are also available for use with minikube such as: virtualbox/vmwarefusion/kvm/xhyve/hyperv.

    Follow [instructions here](https://blog.alexellis.io/kvm-kubernetes-primer/) to install packages from `apt`.

    @@ -128,4 +130,8 @@ Find the IP of the minikube VM with `echo $(minikube ip --profile cluster1)` i.e
    ssh -L 31112:192.168.39.10:31112 user@linux-host
    ```

    You can now access the OpenFaaS installation from your remote Linux host via http://127.0.0.1:31112 or even using `faas-cli --gateway 127.0.0.1:31112`.
    You can now access the OpenFaaS installation from your remote Linux host via http://127.0.0.1:31112 or even using `faas-cli --gateway 127.0.0.1:31112`.

    ### Wrapping up

    If you have comments, questions or suggestions feel free to reach out over Twitter [@alexellisuk](https://twitter.com/alexellisuk)
  10. @alexellis alexellis revised this gist Aug 21, 2018. No changes.
  11. @alexellis alexellis revised this gist Aug 21, 2018. 1 changed file with 24 additions and 11 deletions.
    35 changes: 24 additions & 11 deletions kvm_minikube.md
    Original file line number Diff line number Diff line change
    @@ -1,14 +1,17 @@
    Use Ubuntu 16.04 as a base installation.
    ## Pre-reqs

    You'll need nested virtualization enabled or a bare metal machine in the cloud. You can find these at Packet.net or Scaleway.
    * Use Ubuntu 16.04 as a base installation.
    * You'll need nested virtualization enabled or a bare metal machine in the cloud. You can find these at Packet.net or Scaleway.

    * Install KVM
    ## Install Tooling

    ### Install KVM

    https://www.linux-kvm.org/page/Main_Page

    Follow [instructions here](https://blog.alexellis.io/kvm-kubernetes-primer/) to install packages from `apt`.

    * Install kubectl
    ### Install kubectl

    https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl

    @@ -18,7 +21,7 @@ chmod +x kubectl
    sudo mv kubectl /usr/local/bin/
    ```

    * Install minikube and the KVM2 driver
    ### Install minikube and the KVM2 driver

    ```
    curl -SLO https://github.com/kubernetes/minikube/releases/download/v0.28.2/docker-machine-driver-kvm2
    @@ -31,7 +34,7 @@ sudo mv docker-machine-driver-kvm2 /usr/local/bin
    sudo mv minikube-linux-amd64 /usr/local/bin/minikube
    ```

    * Create a cluster with `kubeadm`
    ## Create a cluster with `kubeadm`

    Using the kubeadm bootstrapper will enable RBAC

    @@ -70,7 +73,7 @@ CURRENT NAME CLUSTER AUTHINFO NAMESPACE
    * cluster1 cluster1 cluster1
    ```

    * Working with the various IPs
    ### Working with the various IPs

    Pass the `--profile` flag to minikube commands so that you can access the ports you want.

    @@ -81,11 +84,11 @@ minikube ip --profile cluster1

    You may want to use SSH port forwarding with `ssh -L port:port [email protected]` or `kubectl port-forward` to give access to services and NodePorts available on the minikube VM. If you're comfortable with `iptables` you could also set up some NAT rules, but I would recommend against it since the IP addresses of your minikube environments may change when being restarted.

    * Deploy something (OpenFaaS)
    ## Deploy something to test it out (OpenFaaS)

    You can now install something to test the cluster.

    Configure helm and tiller:
    * Configure helm and tiller:

    ```
    curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash
    @@ -96,7 +99,7 @@ kubectl -n kube-system create sa tiller \
    helm init --skip-refresh --upgrade --service-account tiller
    ```

    Setup OpenFaaS via helm:
    * Setup OpenFaaS via helm:

    ```
    kubectl apply -f https://raw.githubusercontent.com/openfaas/faas-netes/master/namespaces.yml
    @@ -109,10 +112,20 @@ helm repo update && helm upgrade openfaas \
    --set functionNamespace=openfaas-fn
    ```

    Now access the OpenFaaS gateway via the NodePort:
    * Now access the OpenFaaS gateway via the NodePort:

    ```
    curl http://$(minikube ip --profile cluster1):31112/system/info
    {"provider":{"provider":"faas-netes","version":{"sha":"5539cf43c15a28e9af998cdc25b5da06252b62e1","release":"0.6.0"},"orchestration":"kubernetes"},"version":{"commit_message":"Attach X-Call-Id to asynchronous calls","sha":"c86de503c7a20a46645239b9b081e029b15bf69b","release":"0.8.11"}}
    ```

    Port forward from your laptop to the Linux machine:

    Find the IP of the minikube VM with `echo $(minikube ip --profile cluster1)` i.e. `192.168.39.10`

    ```
    ssh -L 31112:192.168.39.10:31112 user@linux-host
    ```

    You can now access the OpenFaaS installation from your remote Linux host via http://127.0.0.1:31112 or even using `faas-cli --gateway 127.0.0.1:31112`.
  12. @alexellis alexellis revised this gist Aug 21, 2018. 1 changed file with 34 additions and 0 deletions.
    34 changes: 34 additions & 0 deletions kvm_minikube.md
    Original file line number Diff line number Diff line change
    @@ -81,4 +81,38 @@ minikube ip --profile cluster1

    You may want to use SSH port forwarding with `ssh -L port:port [email protected]` or `kubectl port-forward` to give access to services and NodePorts available on the minikube VM. If you're comfortable with `iptables` you could also set up some NAT rules, but I would recommend against it since the IP addresses of your minikube environments may change when being restarted.

    * Deploy something (OpenFaaS)

    You can now install something to test the cluster.

    Configure helm and tiller:

    ```
    curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash
    kubectl -n kube-system create sa tiller \
    && kubectl create clusterrolebinding tiller \
    --clusterrole cluster-admin \
    --serviceaccount=kube-system:tiller
    helm init --skip-refresh --upgrade --service-account tiller
    ```

    Setup OpenFaaS via helm:

    ```
    kubectl apply -f https://raw.githubusercontent.com/openfaas/faas-netes/master/namespaces.yml
    helm repo add openfaas https://openfaas.github.io/faas-netes/
    helm repo update && helm upgrade openfaas \
    --install openfaas/openfaas \
    --namespace openfaas \
    --set functionNamespace=openfaas-fn
    ```

    Now access the OpenFaaS gateway via the NodePort:

    ```
    curl http://$(minikube ip --profile cluster1):31112/system/info
    {"provider":{"provider":"faas-netes","version":{"sha":"5539cf43c15a28e9af998cdc25b5da06252b62e1","release":"0.6.0"},"orchestration":"kubernetes"},"version":{"commit_message":"Attach X-Call-Id to asynchronous calls","sha":"c86de503c7a20a46645239b9b081e029b15bf69b","release":"0.8.11"}}
    ```
  13. @alexellis alexellis revised this gist Aug 21, 2018. 1 changed file with 13 additions and 0 deletions.
    13 changes: 13 additions & 0 deletions kvm_minikube.md
    Original file line number Diff line number Diff line change
    @@ -69,3 +69,16 @@ kubectl config get-contexts
    CURRENT NAME CLUSTER AUTHINFO NAMESPACE
    * cluster1 cluster1 cluster1
    ```

    * Working with the various IPs

    Pass the `--profile` flag to minikube commands so that you can access the ports you want.

    ```
    minikube ip --profile cluster1
    192.168.39.10
    ```

    You may want to use SSH port forwarding with `ssh -L port:port [email protected]` or `kubectl port-forward` to give access to services and NodePorts available on the minikube VM. If you're comfortable with `iptables` you could also set up some NAT rules, but I would recommend against it since the IP addresses of your minikube environments may change when being restarted.


  14. @alexellis alexellis revised this gist Aug 21, 2018. 1 changed file with 9 additions and 1 deletion.
    10 changes: 9 additions & 1 deletion kvm_minikube.md
    Original file line number Diff line number Diff line change
    @@ -60,4 +60,12 @@ Setting up kubeconfig...
    Starting cluster components...
    Kubectl is now configured to use the cluster.
    Loading cached images from config file.
    ```
    ```

    You can manage each cluster using `kubectl` by switching between contexts saved in ~/.kube/config.yaml. The [kubectx](https://github.com/ahmetb/kubectx) tool is also popular in the community for switching between these quickly.

    ```
    kubectl config get-contexts
    CURRENT NAME CLUSTER AUTHINFO NAMESPACE
    * cluster1 cluster1 cluster1
    ```
  15. @alexellis alexellis revised this gist Aug 21, 2018. 1 changed file with 22 additions and 0 deletions.
    22 changes: 22 additions & 0 deletions kvm_minikube.md
    Original file line number Diff line number Diff line change
    @@ -35,7 +35,29 @@ sudo mv minikube-linux-amd64 /usr/local/bin/minikube

    Using the kubeadm bootstrapper will enable RBAC

    Create your first cluster VM:

    ```
    minikube start --bootstrapper=kubeadm --vm-driver=kvm2 --memory 4096 --cpus 4 --profile cluster1
    ```

    You can set up additional separate VMs using the `--profile` flag.

    ```
    Starting local Kubernetes v1.10.0 cluster...
    Starting VM...
    Downloading Minikube ISO
    160.27 MB / 160.27 MB [============================================] 100.00% 0s
    Getting VM IP address...
    Moving files into cluster...
    Downloading kubeadm v1.10.0
    Downloading kubelet v1.10.0
    Finished Downloading kubelet v1.10.0
    Finished Downloading kubeadm v1.10.0
    Setting up certs...
    Connecting to cluster...
    Setting up kubeconfig...
    Starting cluster components...
    Kubectl is now configured to use the cluster.
    Loading cached images from config file.
    ```
  16. @alexellis alexellis revised this gist Aug 21, 2018. 1 changed file with 3 additions and 5 deletions.
    8 changes: 3 additions & 5 deletions kvm_minikube.md
    Original file line number Diff line number Diff line change
    @@ -6,6 +6,8 @@ You'll need nested virtualization enabled or a bare metal machine in the cloud.

    https://www.linux-kvm.org/page/Main_Page

    Follow [instructions here](https://blog.alexellis.io/kvm-kubernetes-primer/) to install packages from `apt`.

    * Install kubectl

    https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl
    @@ -16,11 +18,7 @@ chmod +x kubectl
    sudo mv kubectl /usr/local/bin/
    ```

    * Install minikube

    https://github.com/kubernetes/minikube/releases

    * Get the KVM2 minikube driver
    * Install minikube and the KVM2 driver

    ```
    curl -SLO https://github.com/kubernetes/minikube/releases/download/v0.28.2/docker-machine-driver-kvm2
  17. @alexellis alexellis created this gist Aug 21, 2018.
    43 changes: 43 additions & 0 deletions kvm_minikube.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,43 @@
    Use Ubuntu 16.04 as a base installation.

    You'll need nested virtualization enabled or a bare metal machine in the cloud. You can find these at Packet.net or Scaleway.

    * Install KVM

    https://www.linux-kvm.org/page/Main_Page

    * Install kubectl

    https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl

    ```
    curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
    chmod +x kubectl
    sudo mv kubectl /usr/local/bin/
    ```

    * Install minikube

    https://github.com/kubernetes/minikube/releases

    * Get the KVM2 minikube driver

    ```
    curl -SLO https://github.com/kubernetes/minikube/releases/download/v0.28.2/docker-machine-driver-kvm2
    curl -SLO https://github.com/kubernetes/minikube/releases/download/v0.28.2/minikube-linux-amd64
    chmod +x docker-machine-driver-kvm2
    chmod +x minikube-linux-amd64
    sudo mv docker-machine-driver-kvm2 /usr/local/bin
    sudo mv minikube-linux-amd64 /usr/local/bin/minikube
    ```

    * Create a cluster with `kubeadm`

    Using the kubeadm bootstrapper will enable RBAC

    ```
    minikube start --bootstrapper=kubeadm --vm-driver=kvm2 --memory 4096 --cpus 4 --profile cluster1
    ```