littlecxm · May 3, 2024 15:10 · Nov 22, 2020 · Nov 22, 2020 · Nov 16, 2020 · Nov 16, 2020
diff --git a/README.md b/README.md
@@ -31,6 +31,16 @@ sysctl --system
 ![confirm.png](https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185/raw/confirm.png)
 
 
+## Back on the Host
+
+Edit the config file for the container (`/etc/pve/lxc/$ID.conf`) and add the following:
+```text
+lxc.apparmor.profile: unconfined
+lxc.cgroup.devices.allow: a
+lxc.cap.drop:
+lxc.mount.auto: "proc:rw sys:rw"
+```
+
 ## In the container
 #### /etc/rc.local
 

diff --git a/README.md b/README.md
@@ -1,5 +1,3 @@
-# k3s in LXC on Proxmox
-
 ## On the host
 #### Ensure these modules are loaded
 ```shell

diff --git a/README.md b/README.md
@@ -73,6 +73,13 @@ k3sup install --ip $CONTAINER_IP --user root
 #### Test
 ```shell
 KUBECONFIG=kubeconfig kubectl get pods --all-namespaces
+NAMESPACE     NAME                                     READY   STATUS      RESTARTS   AGE
+kube-system   metrics-server-7566d596c8-zm7tj          1/1     Running     0          69m
+kube-system   local-path-provisioner-6d59f47c7-ldbcl   1/1     Running     0          69m
+kube-system   helm-install-traefik-glt48               0/1     Completed   0          69m
+kube-system   coredns-7944c66d8d-67lxp                 1/1     Running     0          69m
+kube-system   traefik-758cd5fc85-wzcst                 1/1     Running     0          68m
+kube-system   svclb-traefik-cwd9h                      2/2     Running     0          42m
 ```
 
 ## References

diff --git a/README.md b/README.md
@@ -21,16 +21,16 @@ sysctl --system
 
 ## Create the k3s container
 #### Uncheck unprivileged container
-![Screenshot from 2020-11-16 04-15-08.png](:/cd3cbb3c473d42f783790b020077b21a)
+![general.png](https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185/raw/general.png)
 
 #### Set swap to 0
-![Screenshot from 2020-11-16 04-18-37.png](:/3d06672ee1444cf3a485c0598e38ba3d)
+![memory.png](https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185/raw/memory.png)
 
 #### Enable DHCP
-![Screenshot from 2020-11-16 04-16-38.png](:/b64d8e297bd5475e8e38d119c655c237)
+![network.png](https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185/raw/network.png)
 
 #### Results
-![Screenshot from 2020-11-16 04-17-03.png](:/d9e5cd33f2ef428ab9d8064541c39774)
+![confirm.png](https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185/raw/confirm.png)
 
 
 ## In the container

diff --git a/d9e5cd33f2ef428ab9d8064541c39774.png → confirm.png b/d9e5cd33f2ef428ab9d8064541c39774.png → confirm.png
diff --git a/cd3cbb3c473d42f783790b020077b21a.png → general.png b/cd3cbb3c473d42f783790b020077b21a.png → general.png
diff --git a/3d06672ee1444cf3a485c0598e38ba3d.png → memory.png b/3d06672ee1444cf3a485c0598e38ba3d.png → memory.png
diff --git a/b64d8e297bd5475e8e38d119c655c237.png → network.png b/b64d8e297bd5475e8e38d119c655c237.png → network.png
diff --git a/3d06672ee1444cf3a485c0598e38ba3d.png b/3d06672ee1444cf3a485c0598e38ba3d.png
diff --git a/b64d8e297bd5475e8e38d119c655c237.png b/b64d8e297bd5475e8e38d119c655c237.png
diff --git a/cd3cbb3c473d42f783790b020077b21a.png b/cd3cbb3c473d42f783790b020077b21a.png
diff --git a/d9e5cd33f2ef428ab9d8064541c39774.png b/d9e5cd33f2ef428ab9d8064541c39774.png
diff --git a/README.md b/README.md
@@ -76,4 +76,6 @@ KUBECONFIG=kubeconfig kubectl get pods --all-namespaces
 ```
 
 ## References
-https://github.com/MarijnKoesen/kubernetes-in-proxmox-with-kubeadm-lxc-and-wireshark
+- https://github.com/MarijnKoesen/kubernetes-in-proxmox-with-kubeadm-lxc-and-wireshark
+- https://github.com/kubernetes-sigs/kind/issues/662
+- https://medium.com/@kvaps/run-kubernetes-in-lxc-container-f04aa94b6c9c
diff --git a/README.md b/README.md
@@ -0,0 +1,79 @@
+# k3s in LXC on Proxmox
+
+## On the host
+#### Ensure these modules are loaded
+```shell
+cat /proc/sys/net/bridge/bridge-nf-call-iptables
+```
+
+#### Disable swap
+```shell
+sysctl vm.swappiness=0
+swapoff -a
+```
+
+#### Enable IP Forwarding
+The first time I tried to get this working, once the cluster was up, the traefik pods were in CrashloopBackoff due to ip_forwarding being disabled. Since LXC containers share the host's kernel, we need to enable this on the host.
+```
+echo 'net.ipv4.ip_forward=1' >> /etc/sysctl.conf
+sysctl --system
+```
+
+## Create the k3s container
+#### Uncheck unprivileged container
+![Screenshot from 2020-11-16 04-15-08.png](:/cd3cbb3c473d42f783790b020077b21a)
+
+#### Set swap to 0
+![Screenshot from 2020-11-16 04-18-37.png](:/3d06672ee1444cf3a485c0598e38ba3d)
+
+#### Enable DHCP
+![Screenshot from 2020-11-16 04-16-38.png](:/b64d8e297bd5475e8e38d119c655c237)
+
+#### Results
+![Screenshot from 2020-11-16 04-17-03.png](:/d9e5cd33f2ef428ab9d8064541c39774)
+
+
+## In the container
+#### /etc/rc.local
+
+/etc/rc.local doesn't exist in the default 20.04 LXC template provided by Rroxmox. Create it with these contents:
+```sh
+#!/bin/sh -e
+
+# Kubeadm 1.15 needs /dev/kmsg to be there, but it's not in lxc, but we can just use /dev/console instead
+# see: https://github.com/kubernetes-sigs/kind/issues/662
+if [ ! -e /dev/kmsg ]; then
+    ln -s /dev/console /dev/kmsg
+fi
+
+# https://medium.com/@kvaps/run-kubernetes-in-lxc-container-f04aa94b6c9c
+mount --make-rshared /
+```
+
+Then run this:
+```shell
+chmod +x /etc/rc.local
+reboot
+```
+
+
+## Installing k8s
+#### k3sup Installation
+Assuming $HOME/bin is in your PATH:
+```shell
+curl -sLS https://get.k3sup.dev | sh
+mv k3sup ~/bin/k3sup && chmod +x ~/bin/k3sup
+```
+
+#### k8s Installation
+```shell
+k3sup install --ip $CONTAINER_IP --user root
+```
+
+#### Test
+```shell
+KUBECONFIG=kubeconfig kubectl get pods --all-namespaces
+```
+
+## References
+https://github.com/MarijnKoesen/kubernetes-in-proxmox-with-kubeadm-lxc-and-wireshark