Skip to content

Instantly share code, notes, and snippets.

@davidandreoletti

davidandreoletti/linux-vms-on-apple-m1-with-networking.md

Forked from max-i-mil/linux-vms-on-apple-m1-with-networking.md
Created September 16, 2023 14:30
Show Gist options
  • Save davidandreoletti/af2a17ea095af9476ad012b4a2365a40 to your computer and use it in GitHub Desktop.
Save davidandreoletti/af2a17ea095af9476ad012b4a2365a40 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. @max-i-mil max-i-mil revised this gist Nov 9, 2022. 1 changed file with 8 additions and 5 deletions.
    13 changes: 8 additions & 5 deletions linux-vms-on-apple-m1-with-networking.md
    View file
    Original file line number Diff line number Diff line change
    @@ -6,17 +6,19 @@ The aim was to be able to:
    1. Run multiple Linux VMs on an Apple M1/ARM device
    2. Use Apple's HVF for native performance speeds
    3. Configure VMs to allow network access to each other
    4. Configure VMs to allow access the internet
    4. Configure VMs to allow access to the internet
    5. Not rely on custom modifications of software


    I had originally assumed I would be up and running with the above in a few hours, but I was wrong. After looking for examples online, of which there are many. The problem was that they were seemingly all based on either Intel based Apple devices, or for MacOS < 12. I'm using a 2021 MacBookPro with an M1 Pro chip running MacOS 12.6 (Monterey). I have therefore spent a lot of time finding out what doesn't work anymore.

    I was also new to some of the tools I have used including [QEMU](https://www.qemu.org/) and [libvirt](https://libvirt.org/index.html).

    After much experimentation and failure, I have a working example to meet the goals set out. A lot of the issues faced seemed to be that changes to MacOS over the past few years and the time it takes to reflect this in software, has meant a period between what used to work, things not working, and now working again with different implementations.
    After much experimentation and failure, I have a working example to meet the goals set out. A lot of the issues faced seemed to be that changes to MacOS over the past few years and the time it takes to reflect this in software, has meant a period between what used to work, things not working, and now working again with different implementations. The end result achieved using the process defined below looks similar to the following:

    Many examples online to help meet goal 3 relied on [TAP devices](https://en.wikipedia.org/wiki/TUN/TAP). As I understand these were not natively supported by MacOS and required kernel extensions (kexts) to enable the functionality - commonly through a tool called 'tuntaposx'. [As stated on their website](https://tuntaposx.sourceforge.net/) the project has not been maintained since 2015 and references the [requirement from Apple](https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution) to use notarization on all updated or new kernel extensions which tuntaposx does not have. [Tunnelblick details the situation here.](https://tunnelblick.net/cTunTapConnections.html)
    ![image](https://user-images.githubusercontent.com/36989775/200905266-136cc6df-9b1d-44ae-8a57-35b97035934e.png)

    Many examples I found online to help meet goal 3 relied on [TAP devices](https://en.wikipedia.org/wiki/TUN/TAP). As I understand these were not natively supported by MacOS and required kernel extensions (kexts) to enable the functionality - commonly through a tool called 'tuntaposx'. [As stated on their website](https://tuntaposx.sourceforge.net/) the project has not been maintained since 2015 and references the [requirement from Apple](https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution) to use notarization on all updated or new kernel extensions which tuntaposx does not have. [Tunnelblick details the situation here.](https://tunnelblick.net/cTunTapConnections.html)

    There are other helper tools available from QEMU/libvirt to deal with networking like qemu-bridge-helper / libvirt virtual networks - but they too don't yet seem to be working on my device for various reasons. Some of the [current open issues](https://gitlab.com/libvirt/libvirt/-/issues/75) from these projects detail more

    @@ -235,6 +237,7 @@ Each VM can now ping the other VMs that share the same bridge (all started with

    If you pass `--network none` to `virt-install` then you will not have the enp1s0 device attached within your VM and it will have no internet access, but it will still be able to access the other VMs using eth1

    ---
    ## Result

    With everything configured as above the setup allows the following:
    @@ -320,8 +323,8 @@ Trying to run the same in a new terminal resulted in it hanging again, so runnin

    ## Follow On

    Now that this is working, to make it more usable I would like to:
    Now that this is working, my next steps to make it more usable are:

    - Automate the installer, or see if disk images can be used
    - Automate the OS installer, or see if disk images can be used
    - Configure domains via XML template
    - Wrap it with Vagrant to provision everything in one go
  2. @max-i-mil max-i-mil revised this gist Nov 9, 2022. 1 changed file with 41 additions and 0 deletions.
    41 changes: 41 additions & 0 deletions linux-vms-on-apple-m1-with-networking.md
    View file
    Original file line number Diff line number Diff line change
    @@ -235,6 +235,47 @@ Each VM can now ping the other VMs that share the same bridge (all started with

    If you pass `--network none` to `virt-install` then you will not have the enp1s0 device attached within your VM and it will have no internet access, but it will still be able to access the other VMs using eth1

    ## Result

    With everything configured as above the setup allows the following:

    Access VM from Host
    ```sh
    # Run from Apple host
    user@apple-host$ ping 192.168.64.2
    PING 192.168.64.2 (192.168.64.2): 56 data bytes
    64 bytes from 192.168.64.2: icmp_seq=0 ttl=64 time=1.713 ms

    --- 192.168.64.2 ping statistics ---
    1 packets transmitted, 1 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 1.713/1.713/1.713/0.000 ms
    ```

    Access VM from VM
    ```sh
    # Run from VM1 with IP 192.168.64.2
    [root@localhost ~] ping 192.168.64.3
    PING 192.168.64.3 (192.168.64.3) 56(84) bytes of data.
    64 bytes from 192.168.64.3: icmp_seq=1 ttl=64 time=2.34 ms

    --- 192.168.64.3 ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 2.342/2.342/2.342/0.000 ms
    ```

    Access Host from VM
    ```sh
    # Run from VM1
    [root@localhost ~] ping 10.0.2.2
    PING 10.0.2.2 (10.0.2.2) 56(84) bytes of data.
    64 bytes from 10.0.2.2: icmp_seq=1 ttl=255 time=0.868 ms

    --- 10.0.2.2 ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 0.868/0.868/0.868/0.000 ms
    ```


    ## Fixes

    Some errors I encountered along the way but not sure if they would be consistent for everyone
  3. @max-i-mil max-i-mil revised this gist Nov 9, 2022. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion linux-vms-on-apple-m1-with-networking.md
    View file
    Original file line number Diff line number Diff line change
    @@ -12,7 +12,7 @@ The aim was to be able to:

    I had originally assumed I would be up and running with the above in a few hours, but I was wrong. After looking for examples online, of which there are many. The problem was that they were seemingly all based on either Intel based Apple devices, or for MacOS < 12. I'm using a 2021 MacBookPro with an M1 Pro chip running MacOS 12.6 (Monterey). I have therefore spent a lot of time finding out what doesn't work anymore.

    I was also new to some of the tools I have used including QEMU [TODO link] and libvirt [TODO link].
    I was also new to some of the tools I have used including [QEMU](https://www.qemu.org/) and [libvirt](https://libvirt.org/index.html).

    After much experimentation and failure, I have a working example to meet the goals set out. A lot of the issues faced seemed to be that changes to MacOS over the past few years and the time it takes to reflect this in software, has meant a period between what used to work, things not working, and now working again with different implementations.

  4. @max-i-mil max-i-mil created this gist Nov 9, 2022.
    286 changes: 286 additions & 0 deletions linux-vms-on-apple-m1-with-networking.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,286 @@
    # Linux Virtual Machines with Private Network on an Apple M1 Device

    ## Background

    The aim was to be able to:
    1. Run multiple Linux VMs on an Apple M1/ARM device
    2. Use Apple's HVF for native performance speeds
    3. Configure VMs to allow network access to each other
    4. Configure VMs to allow access the internet
    5. Not rely on custom modifications of software


    I had originally assumed I would be up and running with the above in a few hours, but I was wrong. After looking for examples online, of which there are many. The problem was that they were seemingly all based on either Intel based Apple devices, or for MacOS < 12. I'm using a 2021 MacBookPro with an M1 Pro chip running MacOS 12.6 (Monterey). I have therefore spent a lot of time finding out what doesn't work anymore.

    I was also new to some of the tools I have used including QEMU [TODO link] and libvirt [TODO link].

    After much experimentation and failure, I have a working example to meet the goals set out. A lot of the issues faced seemed to be that changes to MacOS over the past few years and the time it takes to reflect this in software, has meant a period between what used to work, things not working, and now working again with different implementations.

    Many examples online to help meet goal 3 relied on [TAP devices](https://en.wikipedia.org/wiki/TUN/TAP). As I understand these were not natively supported by MacOS and required kernel extensions (kexts) to enable the functionality - commonly through a tool called 'tuntaposx'. [As stated on their website](https://tuntaposx.sourceforge.net/) the project has not been maintained since 2015 and references the [requirement from Apple](https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution) to use notarization on all updated or new kernel extensions which tuntaposx does not have. [Tunnelblick details the situation here.](https://tunnelblick.net/cTunTapConnections.html)

    There are other helper tools available from QEMU/libvirt to deal with networking like qemu-bridge-helper / libvirt virtual networks - but they too don't yet seem to be working on my device for various reasons. Some of the [current open issues](https://gitlab.com/libvirt/libvirt/-/issues/75) from these projects detail more

    ---

    ## Requirements

    I have tested this using:

    - MacOS - 12.6.1 Monterey
    - Chip - M1 Pro
    - QEMU - 7.1.0
    - libvirt - 8.9.0
    - virt-manager - 4.1.0

    As it's MacOS, I have used homebrew for package management. For convenience I am also using [brew services](https://docs.brew.sh/Manpage#services-subcommand) as a wrapper for launchctl

    ```sh
    brew install qemu libvirt virt-manager

    # must use sudo as it affects qemu later
    sudo brew services start libvirt
    ```

    For native speeds you will need an OS ISO image for AARCH64 hosts (ARM). x86 ISO files will work but because Apple M1 devices use an ARM architecture they will need QEMU to emulate the necessary hardware which is much slower, and you may need to alter the command given later to use 'tcg' instead of 'hvf'.

    I went with RHEL 9 as [RHEL 7/8 wont work natively due to page sizes](https://access.redhat.com/solutions/6545411)

    Basic libvirt + QEMU knowledge is also required.

    The working solution relies on the latest version of QEMU (7.1.0 as of Nov 2022). Some of the functionality has been added, with the man page updated, **but with the QEMU wiki docs online not yet featuring the same detail on the invocation page**.

    ---
    ## Process

    With dependencies installed - you can now create a VM using a single command, and everything else will be handled automatically. You can run the same command again to create a second VM.

    ```sh
    # create a libvirt 'domain' - specify some options using qemu command line pass through
    sudo virt-install \
    --name host1 \
    --memory 2048 \
    --vcpus 2 \
    --disk size=30 \
    --cdrom /path/to/ISO_files/rhel-baseos-9.0-aarch64-dvd.iso \
    --os-variant rhel9.0 \
    --virt-type hvf \
    --qemu-commandline='-M highmem=off -netdev vmnet-shared,id=net0 -device virtio-net-device,netdev=net0,mac=54:54:00:55:54:51' \
    --network user

    # create a second libvirt 'domain' - ensure the friendly name and MAC address of the net device are unique and run the same command again
    sudo virt-install \
    --name host2 \
    --memory 2048 \
    --vcpus 2 \
    --disk size=30 \
    --cdrom /path/to/ISO_files/rhel-baseos-9.0-aarch64-dvd.iso \
    --os-variant rhel9.0 \
    --virt-type hvf \
    --qemu-commandline='-M highmem=off -netdev vmnet-shared,id=net0 -device virtio-net-device,netdev=net0,mac=54:54:00:55:54:52' \
    --network user

    # create Nth libvirt domain - just keep name and MAC address unique
    sudo virt-install \
    --name hostN \
    --memory 2048 \
    --vcpus 2 \
    --disk size=30 \
    --cdrom /path/to/ISO_files/rhel-baseos-9.0-aarch64-dvd.iso \
    --os-variant rhel9.0 \
    --virt-type hvf \
    --qemu-commandline='-M highmem=off -netdev vmnet-shared,id=net0 -device virtio-net-device,netdev=net0,mac=NN:NN:NN:NN:NN:NN' \
    --network user
    ```

    Notable flags are:

    > `--virt-type hvf` *tell QEMU to use the Apple Hypervisor Framework instead of an alternative such as TCG. This enables the native speeds*
    > `--network user` *enable QEMU user mode networking, which gives us access to the host and the internet. Alternative is 'none' which removes access to both. Other named libvirt networks do not seem to work on the current version for my device i.e. '--network name=default'*
    >From the `--qemu-commandline` flag
    >> `-netdev vmnet-shared` *tells QEMU to use the [vmnet APIs provided by Apple](https://developer.apple.com/documentation/vmnet) as part of HVF. This is the alternative to managing our own bridge/TAP devices commonly used in the past. Flag alternatives are [vmnet-host and vmnet-bridged](https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html?#qapidoc-1408)*
    >> `-device mac=XX:YY...` *defines the MAC address used for the virtual device that will be attached to our VM. Must be unique per VM for DHCP*
    >>
    > I wasn't able to find virt-install equivalents for 'vmnet-*' netdev options yet hence the qemu-commandline workaround but this may come in a future release
    In my case I connected to each VM via the console to complete the installation process before continuing.

    This has now done a few things in the background:

    ### On the Apple host:

    - A bridge device has been created called 'bridge100' (sequential from 100 if you start multiple vmnet types at once)
    - A vmenet device has been created for each VM requested, and then attached to the bridge
    - An internal macOS DHCP service (bootpd) has been started to assign IP addresses to each VM (and the bridge)
    - The QEMU [user mode network](https://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29) (SLIRP) has been created which gives internet access over the default gateway 10.0.2.2

    ```sh
    # some output omitted
    root@apple-host$ ifconfig
    ...
    vmenet0: flags=8b63<UP,BROADCAST,SMART,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
    ether aa:70:70:52:86:d4
    media: autoselect
    status: active
    # bridge100 has IP 192.168.64.1 assigned
    # vmenet0 and vmenet1 are members
    bridge100: flags=8a63<UP,BROADCAST,SMART,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
    options=3<RXCSUM,TXCSUM>
    ether be:d0:74:61:f6:64
    inet 192.168.64.1 netmask 0xffffff00 broadcast 192.168.64.255
    ...
    Configuration:
    ...
    member: vmenet0 flags=3<LEARNING,DISCOVER>
    ifmaxaddr 0 port 29 priority 0 path cost 0
    member: vmenet1 flags=3<LEARNING,DISCOVER>
    ifmaxaddr 0 port 31 priority 0 path cost 0
    ...
    status: active
    vmenet1: flags=8b63<UP,BROADCAST,SMART,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
    ether b6:69:a5:54:74:45
    media: autoselect
    status: active
    ```

    ```xml
    <!-- Contents of /etc/bootpd.plist after creating VMs
    dhcp_domain_name_server is same IP as bridge100-->
    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
    <plist version="1.0">
    <dict>
    <key>Subnets</key>
    <array>
    <dict>
    <key>_creator</key>
    <string>com.apple.NetworkSharing</string>
    <key>allocate</key>
    <true/>
    <key>dhcp_domain_name_server</key>
    <array>
    <string>192.168.64.1</string>
    </array>
    <key>dhcp_router</key>
    <string>192.168.64.1</string>
    <key>interface</key>
    <string>bridge100</string>
    <key>lease_max</key>
    <integer>86400</integer>
    <key>lease_min</key>
    <integer>86400</integer>
    <key>name</key>
    <string>192.168.64/24</string>
    <key>net_address</key>
    <string>192.168.64.0</string>
    <key>net_mask</key>
    <string>255.255.255.0</string>
    <key>net_range</key>
    <array>
    <string>192.168.64.2</string>
    <string>192.168.64.254</string>
    </array>
    </dict>
    </array>
    ...
    ```
    ### On the guest Linux VMs:

    ```sh
    # IP config of first VM - some output omitted

    # enp1s0 is on the QEMU SLIRP network and has access to host + internet
    # eth1 is attached to bridge100 and has IP 192.160.64.2 assigned via DHCP - has access via bridge to other VMs we create
    # eth1 MAC address matches that defined by 'virt-install'
    [root@localhost ~] ip a s
    2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:d1:d1:8d brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute enp1s0
    valid_lft 84794sec preferred_lft 84794sec
    ...
    3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 54:54:00:55:54:51 brd ff:ff:ff:ff:ff:ff
    inet 192.168.64.2/24 brd 192.168.64.255 scope global dynamic noprefixroute eth1
    valid_lft 84794sec preferred_lft 84794sec
    ...
    ```

    ```sh
    # IP config of second VM - some output omitted

    # enp1s0 has same IP as VM1 as the SLIRP network for each VM is independent so it isn't clashing
    # eth1 has IP 192.168.64.3 assigned
    [root@localhost ~] ip a s
    2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:ae:7e:5c brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute enp1s0
    valid_lft 85064sec preferred_lft 85064sec
    ...
    3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 54:54:00:55:54:52 brd ff:ff:ff:ff:ff:ff
    inet 192.168.64.3/24 brd 192.168.64.255 scope global dynamic noprefixroute eth1
    valid_lft 85064sec preferred_lft 85064sec
    ...
    ```
    ```sh
    # Routing tables used by both VMs
    [root@localhost ~] ip route
    default via 10.0.2.2 dev enp1s0 proto dhcp metric 100
    default via 192.168.64.1 dev eth1 proto dhcp metric 101
    10.0.2.0/24 dev enp1s0 proto kernel scope link src 10.0.2.15 metric 100
    192.168.64.0/24 dev eth1 proto kernel scope link src <host_specific_IP> metric 101
    ```

    Each VM can now ping the other VMs that share the same bridge (all started with the same vmnet parameter). Each VM can also make outbound internet connections via the SLIRP network.

    If you pass `--network none` to `virt-install` then you will not have the enp1s0 device attached within your VM and it will have no internet access, but it will still be able to access the other VMs using eth1

    ## Fixes

    Some errors I encountered along the way but not sure if they would be consistent for everyone

    ### 1 - Add default URI for libvirt connections

    When running `virt-install`

    Error:
    > error: failed to connect to the hypervisor
    > error: Operation not supported: Cannot use direct socket mode if no URI is set
    Fix:
    ```sh
    echo 'uri = qemu:///system' >> /opt/homebrew/etc/libvirt/libvirt.conf

    brew services restart libvirt
    ```

    This provides the default URI to connections to QEMU system instances

    ### 2 - Manually start virtlogd

    When starting libvirt - the expectation was that libvirt would manage virtlogd, which is used to manage logs from VMs as an alternative to writing them all to a file which need manual disk management.

    Error:
    > error: failed to connect to the hypervisor
    > error: Failed to connect socket to '/opt/homebrew/var/run/libvirt/virtlogd-sock': No such file or directory
    Fix:
    ```sh
    sudo /opt/homebrew/sbin/virtlogd -d
    ```

    This will start virlodg as a daemon process manually

    ### 3 - Hanging when running `virt-install`

    I have seen some inconsistent behaviour when running virt-install which means the command hangs. I have had to send 'Ctrl + C' - wait for the process to exit (which can take up to a few minutes) then re-run the same command a second time and it works without issue.

    Trying to run the same in a new terminal resulted in it hanging again, so running twice in the same session was what worked for me.

    ## Follow On

    Now that this is working, to make it more usable I would like to:

    - Automate the installer, or see if disk images can be used
    - Configure domains via XML template
    - Wrap it with Vagrant to provision everything in one go