Created
September 29, 2023 20:48
-
-
Save Ravenstine/707180ef29e9d37a8f816e019ca32dbf to your computer and use it in GitHub Desktop.
Revisions
-
Ravenstine created this gist
Sep 29, 2023 .There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,321 @@ How To Run Yggdrasil In Docker ============================== Want to run Yggdrasil in a Docker container? This is how you can do it. The keys to getting it working are the following: - Give the container access to the TUN interface of the host (or the VM guest in the case of Docker Machine or Docker for Mac) - Enable IPv6 - Assign a dedicated MAC address It took me a while to figure this stuff out, so hopefully this helps you. When the requirements aren't met, Yggdrasil exits with a barely helpful "permission denied" message. Here is an example Dockerfile: ```dockerfile FROM alpine:3.18.3 RUN \ apk add \ --no-cache \ --no-check-certificate \ --allow-untrusted \ --no-scripts \ yggdrasil=0.4.7-r9 CMD yggdrasil -useconffile /etc/yggdrasil.conf ``` At this point, you will likely want to create a yggdrasil.conf file that configures the host machine as a peer. Assuming that your host is running Yggdrasil and listening for peers on TCP port 8008, this is how you can generate a new yggdrasil.conf file for your container: ```sh echo "{ \"Peers\": [ \"tcp://host.docker.internal:8008\" ] }" | yggdrasil -useconf -normaliseconf > yggdrasil.conf ``` The specific port number doesn't matter so long as it's the one that the peer running on your host is listening for other peers on. The `host.docker.internal` domain is automatically set up by Docker. Here is an example of how to run a container: ```sh docker build -t yggdrasil . docker run \ --rm \ --cap-add "NET_ADMIN" \ --device "/dev/net/tun" \ --volume "./yggdrasil.conf:/etc/yggdrasil.conf" \ --sysctl "net.ipv6.conf.all.disable_ipv6=0" \ --mac-address "1E:FC:B2:8D:DF:D4" \ yggdrasil ``` Here is an example with Docker Compose: ```yaml version: '3.9' services: yggdrasil: build: . cap_add: - NET_ADMIN devices: - /dev/net/tun volumes: - "./yggdrasil.conf:/etc/yggdrasil.conf" sysctls: - "net.ipv6.conf.all.disable_ipv6=0" mac_address: 1E:FC:B2:8D:DF:D4 ``` In which case, you would run `docker-compose up`. To spin it down, run `docker-compose down`. Check the log of the running container to see whether Yggdrasil is successfully reaching the host peer. If you are unsure, you can check on your host machine by running `yggdrasilctl getPeers`. Ok, this is all fine, but now you want to make another containerized service available from your Yggdrasil peer container, right? The following is an approach using Docker Compose, with a Redis service as an example. We will need to change the Dockerfile so that it will support forwarding the Redis port. ```dockerfile FROM alpine:3.18.3 RUN \ apk add \ --no-cache \ --no-check-certificate \ --allow-untrusted \ --no-scripts \ yggdrasil=0.4.7-r9 \ socat=1.7.4.4-r1 \ supervisor=4.2.5-r2 CMD supervisord -c /etc/supervisord.conf ``` We will also need a supervisord.conf file: ```ini [supervisord] logfile=/dev/null nodaemon=true user=root [program:yggdrasil] command=yggdrasil -useconffile /etc/yggdrasil/yggdrasil.conf stdout_logfile=/dev/stdout stdout_logfile_maxbytes=0 stderr_logfile=/dev/stderr stderr_logfile_maxbytes=0 [program:redis-forwarder] command=socat TCP6-LISTEN:6379,fork,forever,reuseaddr TCP4:redis:6379 ``` And now let's add a volume for the supervisord configuration: ```yaml version: '3.9' services: yggdrasil: build: . cap_add: - NET_ADMIN devices: - /dev/net/tun volumes: - "./yggdrasil.conf:/etc/yggdrasil.conf" - "./supervisord.conf:/etc/supervisord.conf" sysctls: - "net.ipv6.conf.all.disable_ipv6=0" mac_address: 1E:FC:B2:8D:DF:D4 redis: image: docker.io/redis:7.2.1-alpine3.18 ports: - "6379:6379" ``` Run `docker-compose up` and use `nc -v <yggdrasil peer ipv6 address> 6379` or `redis-cli -h <yggdrasil peer ipv6 address>` to confirm that your containerized peer has joined your network and that you can reach the Redis service through it. ## Alternative Setup (if all else fails) Can't get IPv6 to work in Docker? `/dev/net/tun` doesn't exist? There's a less optimal alternative that can work, and that's to run a virtual machine within a Docker container. The reason this works is that it emulates a full network stack, bypassing any limitations on the host. Here is an example alternative Dockerfile that runs Yggdrasil within a VM: ```dockerfile FROM alpine:3.18.3 AS initramfs WORKDIR / RUN \ apk add --no-cache \ coreutils \ curl RUN mkdir -p /root ENV initramfsDir=/initramfs ## Make default directory structure RUN \ mkdir -p \ "$initramfsDir" \ "$initramfsDir/bin" \ "$initramfsDir/etc" \ "$initramfsDir/mnt" \ "$initramfsDir/proc" \ "$initramfsDir/root" \ "$initramfsDir/sbin" \ "$initramfsDir/sys" \ "$initramfsDir/var/run" # Add base system RUN \ apk add \ --root "$initramfsDir" \ --no-cache \ --initdb \ --no-check-certificate \ --allow-untrusted \ --repository "https://dl-cdn.alpinelinux.org/alpine/v3.18/main" \ busybox=1.36.1-r2 \ supervisor \ socat # Add yggdrasil RUN \ apk add \ --root "$initramfsDir" \ --initdb \ --no-cache \ --no-check-certificate \ --allow-untrusted \ --no-scripts \ --repository "https://dl-cdn.alpinelinux.org/alpine/v3.18/main" \ --repository "https://dl-cdn.alpinelinux.org/alpine/v3.18/community" \ yggdrasil=0.4.7-r9 ENV linuxLtsDir=/linux-virt RUN mkdir -p $linuxLtsDir RUN \ curl -o linux-virt.tar -sSL https://dl-cdn.alpinelinux.org/alpine/v3.18/main/aarch64/linux-virt-6.1.54-r0.apk && \ tar xvf linux-virt.tar -C $linuxLtsDir && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.alias $initramfsDir/lib/modules/6.1.54-0-virt/modules.alias && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.alias.bin $initramfsDir/lib/modules/6.1.54-0-virt/modules.alias.bin && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.builtin $initramfsDir/lib/modules/6.1.54-0-virt/modules.builtin && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.builtin.alias.bin $initramfsDir/lib/modules/6.1.54-0-virt/modules.builtin.alias.bin && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.builtin.bin $initramfsDir/lib/modules/6.1.54-0-virt/modules.builtin.bin && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.builtin.modinfo $initramfsDir/lib/modules/6.1.54-0-virt/modules.builtin.modinfo && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.dep $initramfsDir/lib/modules/6.1.54-0-virt/modules.dep && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.dep.bin $initramfsDir/lib/modules/6.1.54-0-virt/modules.dep.bin && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.devname $initramfsDir/lib/modules/6.1.54-0-virt/modules.devname && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.order $initramfsDir/lib/modules/6.1.54-0-virt/modules.order && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.softdep $initramfsDir/lib/modules/6.1.54-0-virt/modules.softdep && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.symbols $initramfsDir/lib/modules/6.1.54-0-virt/modules.symbols && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.symbols.bin $initramfsDir/lib/modules/6.1.54-0-virt/modules.symbols.bin && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/drivers/net/ethernet/intel/e1000/e1000.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/drivers/net/ethernet/intel/e1000/e1000.ko.gz && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/net/9p/9pnet.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/net/9p/9pnet.ko.gz && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/net/packet/af_packet.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/net/packet/af_packet.ko.gz && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/drivers/net/tun.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/drivers/net/tun.ko.gz && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/fs/9p/9p.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/fs/9p/9p.ko.gz && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/net/9p/9pnet_virtio.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/net/9p/9pnet_virtio.ko.gz && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/fs/fscache/fscache.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/fs/fscache/fscache.ko.gz && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/fs/netfs/netfs.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/fs/netfs/netfs.ko.gz && \ install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/net/ipv6/ipv6.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/net/ipv6/ipv6.ko.gz ## Add host nameserver RUN cat > $initramfsDir/etc/resolv.conf <<EOF nameserver 10.0.2.3 EOF ## Add init file for initramfs RUN cat > $initramfsDir/init <<EOF #!/bin/sh mount -t proc none /proc mount -t sysfs none /sys mount -t devtmpfs dev /dev echo /sbin/mdev > /proc/sys/kernel/hotplug /sbin/mdev -s # Shared Directory modprobe virtio_pci modprobe 9pnet modprobe 9pnet_virtio modprobe 9p mkdir -p /etc/yggdrasil mount -t 9p -o trans=virtio host0 /etc/yggdrasil # Networking modprobe e1000 # modprobe virtio_net modprobe tun modprobe ipv6 echo "nameserver 10.0.2.3" > /etc/resolv.conf ip link set lo up ip link set eth0 up udhcpc -i eth0 ifconfig eth0 10.0.2.15 route add default gw 10.0.2.2 eth0 mkdir -p /var/run socat TCP6-LISTEN:6379,fork,forever,reuseaddr TCP4:redis:6379 & /usr/bin/yggdrasil -useconffile /etc/yggdrasil/yggdrasil.conf # exec /bin/busybox sh EOF RUN chmod +x "$initramfsDir/init" ## Bundle initramfs file and copy vmlinuz RUN \ cd $initramfsDir && \ find . | sort | cpio --quiet --renumber-inodes -o -H newc | gzip -9 > /root/initramfs && \ cp "$linuxLtsDir/boot/vmlinuz-virt" "/root/vmlinuz" FROM alpine:3.18.3 AS primary COPY --from=initramfs /root /root WORKDIR /root RUN apk add --no-cache \ qemu-system-aarch64 CMD qemu-system-aarch64 \ -m "128M" \ -machine "virt" \ -cpu "cortex-a76" \ -smp "1" \ -kernel "vmlinuz" \ -initrd "initramfs" \ -serial "mon:stdio" \ -append "ip=dhcp" \ -device "e1000,mac=1E:FC:B2:8D:DF:D4,netdev=net0" \ -netdev "user,id=net0" \ -virtfs "local,path=/etc/yggdrasil,mount_tag=host0,security_model=passthrough,id=host0" \ -nographic ``` This is just an example. It's not as efficient because it emulates a CPU, but I believe it's workable if there's something about the host's Docker or network setup that makes it impossible to run Yggdrasil the normal way.