# Custom Boot Objects in Virtualization Framework macOS Guest VMs

By default, "custom boot objects" (created using `kmutil create`)
can't be used in macOS Virtualization framework guest VMs. Here I show
how to get around this Apple design flaw. Note that Virtualization
framework macOS guests can only be created on Apple Silicon Macs.

To proceed, you'll need to work through my [Running Third Party Kernel
Extensions on Virtualization Framework macOS Guest
VMs](https://gist.github.com/steven-michaud/fda019a4ae2df3a9295409053a53a65c). Many
of the steps you need to perform here come from that document: You
need to make all the changes it describes to "stage 0", "stage 1" and
"stage 2" iBoot modules. But you won't (necessarily) need to patch the
kernel cache (`kernelcache`).

Custom boot objects are useful if you need to add kernel extensions
to, or remove them from, the standard set of kernel extensions built
into the kernel cache (the boot kernel collection). You'd also use
them if you were building and installing your own kernel from source,
as per [Building XNU for macOS 11.2 (Intel + Apple
Silicon)](https://kernelshaman.blogspot.com/2021/02/building-xnu-for-macos-112-intel-apple.html). Of
course this is much more difficult, and I won't be covering it here.

The only requirement for building a custom boot object (as of macOS
13) it that you install (in your guest VM) a Kernel Debug Kit for that
version of macOS. Apple doesn't provide KDKs for all versions of
macOS, so you'll need to install a version of macOS, in your guest VM,
for which a KDK is available.

The standard procedure for custom boot objects is to create one using
`kmutil create`, then boot into the recovery partition and run `kmutil
configure-boot` to "install it". With the correct KDK, `kmutil create`
works even if you haven't patched anything in your guest VM. And
`kmutil configure-boot -v /Volumes/Macintosh\ HD -c
custom-kernelcache` appears to work. But the VM always refuses to boot
from the custom kernelcache.

Patching the three iBoot modules allows `kmutil configure-boot` to
work properly for most purposes.

But if you want to load third party kernel extensions, you need to
create the custom kernelcache in a particular way, described
below. Then follow my [Running Third Party Kernel
Extensions](https://gist.github.com/steven-michaud/fda019a4ae2df3a9295409053a53a65c)
document's instructions to patch it and copy it to its final
destination.

Here's a way to create a custom boot object, in `*.im4p` format, that
adds the `AppleARMWatchdogTimer` kext to the guest VM's boot kernel
collection. It assumes the guest VM is running macOS 14.4.

```
kmutil create -a arm64e -z -V release -n boot -B kernelcache.im4p.org --img4-encode -k /Library/Developer/KDKs/KDK_14.4_23E214.kdk/System/Library/Kernels/kernel.release.vmapple -x -b com.apple.driver.AppleARMWatchdogTimer $(kmutil inspect -V release --no-header | awk '{print " -b "$1; }')
```

Make sure you've also found (and copied) the "orginal" `kernelcache`
file, so that you can generate `kernelcache.im4m.org` from it:

- `img4tool -e -m kernelcache.im4m.org kernelcache.org`

Then use the "new" `kernelcache.im4p.org` file in the following
command, and proceed from there.

- `img4tool -e -o kernelcache.bin.org kernelcache.im4p.org`

## More on the AppleARMWatchdogTimer kext

As I say
[here](https://github.com/utmapp/UTM/issues/4026#issuecomment-1341245782),
the failures in `_validate_acm_context()`, which need to be patched in
the kernel cache's `AppleVPBootPolicy` kext, may be caused by a timing
problem. This may be connected to the absence of the
`AppleARMWatchdogTimer` kext from the guest VM's kernel
cache. [Here](https://github.com/utmapp/UTM/issues/4026#issuecomment-1344827407)
and
[here](https://github.com/utmapp/UTM/issues/4026#issuecomment-1707581794)
I talked about trying (and failing) to add this kext to the guest VM's
kernel cache.

Well, I was doing it wrong. The method I describe here works just
fine. But my custom kernel (with `AppleARMWatchdogTimer.kext` added)
still isn't enough, unpatched, to allow loading of third party kexts.

The `AppleARMWatchdogTimer` kext loads automatically on bare metal. It
implements an `IOWatchdog` service, whose client (in userspace) is
`watchdogd`. In userspace an `IOWatchdogUserClient` object is
implemented using a hardware `wdt` device. (See the results of `ioreg
-p IOService -n "IOWatchdogUserClient" -w 0 -r -t` and `ioreg -p
IODeviceTree -n "wdt" -w 0 -r -t`.)

Unlike on bare metal, the `AppleARMWatchdogTimer` kext doesn't load
automatically on a macOS guest VM, even when it's present in the
kernel cache. Both the service client and the device are absent from
userspace. `watchdogd` dies when it can't find the `IOWatchdog`
service:

```
watchdogd: 2977082625: failed to discover watchdog KEXT service
watchdogd: 2977098166: detected virtual machine environment and no watchdog KEXT found, exiting...
```

And, when trying to load a third party kext, `kcgend` complains about
not being able to find the `IOWatchdog` service:

```
kcgend: Could not find IOWatchdog
```

It's possible to load the `AppleARMWatchdogTimer` kext "by hand",
using `kmutil load -b com.apple.driver.AppleARMWatchdogTimer`. Then
you can explicitly load `watchdogd` using `sudo launchctl -p
system/com.apple.watchdogd`. But `watchdogd` doesn't stay loaded. It
logs the "failed to discover watchdog KEXT service" message and dies.

It seems that Apple's Virtualization framework doesn't emulate the
`wdt` hardware. It's not in the guest VM's `devicetree` file, though
it *is* in my Macmini9,1's `devicetree` file. I suspect Apple won't be
able to fix the `_validate_acm_context()` errors without implementing
this device. I can't prove it, but I think the circumstantial evidence
is pretty compelling.

The `devicetree.img4` file exists in the file system. It can be found
the same way as the `iBoot.img4` file. Once you have it, run the
following commands to extract its binary:

```
img4tool -e -p devicetree.im4p devicetree.img4
img4tool -e -o devicetree.bin devicetree.im4p
```

Then run the following command to see its contents in human-readable
format:

```
ipsw dtree devicetree.bin
```

[ipsw](https://github.com/blacktop/ipsw) is available via
[Homebrew](https://brew.sh/). To install it do `brew install ipsw`.

A [devicetree](https://en.wikipedia.org/wiki/Devicetree) describes the
hardware components of a particular computer. It's needed on operating
systems, like macOS on Apple Silicon, that don't have ways to discover
the hardware they're running on.