When adding and removing devices to a QEMU VM created from OSX-KVM, I
encountered some device problems.
While tinkering and fiddling with adding and removing different QEMU devices to
the VM, the order of -device flags passed caused PCI addresses to keep
changing. This, in turn caused keyboard and mouse support to stop working for
me, until I found out what the problem was.
It turns out that enabling debug serial output for the XNU kernel allowed me to track down the issue.
Investigating what OSX-KVM's pre-packaged SSDT files contained, I ran into an
issue with the default USB controllers' PCI addresses.
The USB controllers I was passing to QEMU did not match the default VM definition & SSDT.
To more easily see the XNU kernel debugging output:
- I added a serial console device to the VM via this
libvirtdomain XML snippet:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<!-- ... Original VM definition ... -->
<devices>
<!-- ... other devices ... -->
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<!-- ... rest of VM definition ... -->
</devices>
</domain>- I added the following XNU
boot-args:debug=0x10A -v serial=3 msgbuf=1048576 serialbaud=115200-venables verbose bootingserial=3- Sends debug output over serialmsgbuf=10485760- Sets kern.msgbuf to10485760bytes (10 MiB) to allow a much larger kernel-log.serialbaud=115200- Sets the serial TTY's baud rate (e.g. to connect withscreen /dev/pts/NN 115200)- The
debug=0x10Abit mask enablesDB_LOG_PI_SCRN,DB_SLOG, andDB_KPRTwhich respectively:- Disables graphical panic screen
- Sends some diagnostics to syslog
- Sends kernel's debugging
kprintfto the serial console port - Note: A bitmask can be calculated by a logical OR operation on those values. For example:
0x100 | 0x2 | 0x8 = 0x10A - To add this, I had to:
- First mount the
OpenCore.qcow2disk withqemu-nbdqemu-nbdneedsnbdkernel module loaded:modprobe nbd max_part=8
- Then mount the inner
vfatEFI partition (containing OpenCore bootloader,kexts and ACPI SSDTs) - Edit the
config.plistfile inside - Unmount the EFI partition & disconnect the
qcow2file from/dev/nbd0
- First mount the
- After changing
boot-args, the kernel consolekprintfdebug output can be viewed in:- The
virt-managerwindow under:View -> Consoles -> Serial 1 - By directly connecting to the
/dev/pts/NNdevice with GNU screen:screen /dev/pts/5. - Note: The device number can change, so pay attention that you're connecting to the right one.
This can be seen in
virt-manager's "Details" screen by checking theSerial 1device under "Source path:"
- The
$ sudo qemu-nbd --discard=unmap --detect-zeroes=unmap --connect=/dev/nbd0 ~/src/pub/OSX-KVM/OpenCore/OpenCore.qcow2
$ sudo sgdisk --print /dev/nbd0
Disk /dev/nbd0: 786432 sectors, 384.0 MiB
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 3D196FDE-9679-43B2-B0EA-52513C02D436
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 786398
Partitions will be aligned on 32-sector boundaries
Total free space is 6075 sectors (3.0 MiB)
Number Start (sector) End (sector) Size Code Name
1 2048 300000 145.5 MiB EF00 primary
2 302048 784384 235.5 MiB 8300 primary
$ sudo blkid /dev/nbd0p1
/dev/nbd0p1: SEC_TYPE="msdos" LABEL_FATBOOT="EFI" LABEL="EFI" UUID="BB7E-D63C" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="primary" PARTUUID="d79d1e93-e748-4fef-8de6-fd4e37e3ec5b"
$ mkdir /tmp/oc
$ sudo mount -t vfat /dev/nbd0p1 /tmp/oc
$ vim /tmp/oc/EFI/OC/config.plist
## Search for: boot-args with: /boot-args
## Add the kernel debug boot args:
## debug=0x10A -v serial=3 msgbuf=10485760 serialbaud=115200 keepsyms=1 amfi_get_out_of_my_way=1 tlbto_us=0 vti=9
$ sudo umount /tmp/oc
$ sudo qemu-nbd --disconnect /dev/nbd0
/dev/nbd0 disconnectedFor reference, the original boot-args defined as default by OSX-KVM were:
<key>boot-args</key>
<string>-v keepsyms=1 amfi_get_out_of_my_way=1 tlbto_us=0 vti=9</string>After editing, I ended up with the following boot-args under the <key>NVRAM</key><dict><key>Add</key>... <key>boot-args</key> ... section in config.plist:
<key>boot-args</key>
<string>debug=0x10A -v serial=3 msgbuf=10485760 serialbaud=115200 keepsyms=1 amfi_get_out_of_my_way=1 tlbto_us=0 vti=9</string>Note: For more details on XNU kernel debugging, see here
After booting the kernel, I saw some ACPI table errors in the console log:
ACPI Exception: AE_NOT_FOUND, (SSDT: QEMUUSB) while loading table (20160930/tbxfload-319)
So, for some reason macOS wasn't finding the "QEMUUSB" device at the expected address.
It turns out that the original SSDT in OSX-KVM had defined the PCI address at a different location.
The original binary SSDT-EHCI.aml file inside OpenCore.qcow2 image, under EFI/OC/ACPI/ contained an ACPI table called QEMUUSB.
That QEMUUSB table defined the PCI addresses for the USB controllers, as expected to be attached by QEMU.
However, the problem was that these addresses were not matching what QEMU was giving the devices automatically.
To demystify what's going on, we can use Intel's iasl tool (included in the acpica Arch Linux package).
We can decompile the .aml file and view the source code to find what devices and PCI addresses were defined:
$ cp /tmp/oc/EFI/OC/ACPI/SSDT-EHCI.aml /tmp/
$ iasl /tmp/SSDT-EHCI.aml
Intel ACPI Component Architecture
ASL+ Optimizing Compiler/Disassembler version 20240927
Copyright (c) 2000 - 2023 Intel Corporation
File appears to be binary: found 58 non-ASCII characters, disassembling
Binary file appears to be a valid ACPI table, disassembling
Input file /tmp/SSDT-EHCI.aml, Length 0xA3 (163) bytes
ACPI: SSDT 0x0000000000000000 0000A3 (v01 KGP QEMUUSB 00000000 INTL 20190509)
Pass 1 parse of [SSDT]
Pass 2 parse of [SSDT]
Parsing Deferred Opcodes (Methods/Buffers/Packages/Regions)
Parsing completed
Disassembly completed
ASL Output: /tmp/SSDT-EHCI.dsl - 1232 bytes
### Inspecting the original SSDT for QEMUUSB
$ cat /tmp/SSDT-EHCI.dsl
/*
* Intel ACPI Component Architecture
* AML/ASL+ Disassembler version 20240927 (64-bit version)
* Copyright (c) 2000 - 2023 Intel Corporation
*
* Disassembling to symbolic ASL+ operators
*
* Disassembly of /tmp/SSDT-EHCI.aml
*
* Original Table Header:
* Signature "SSDT"
* Length 0x000000A3 (163)
* Revision 0x01
* Checksum 0xBF
* OEM ID "KGP"
* OEM Table ID "QEMUUSB"
* OEM Revision 0x00000000 (0)
* Compiler ID "INTL"
* Compiler Version 0x20190509 (538510601)
*/
DefinitionBlock ("", "SSDT", 1, "KGP", "QEMUUSB", 0x00000000)
{
External (_SB_.PCI0, DeviceObj)
External (_SB_.PCI0.S38_, DeviceObj)
Scope (\_SB.PCI0)
{
Device (EH01)
{
Name (_ADR, 0x00070007) // _ADR: Address
}
Scope (S38)
{
Name (_STA, Zero) // _STA: Status
}
Device (UHC1)
{
Name (_ADR, 0x00070000) // _ADR: Address
}
Device (UHC2)
{
Name (_ADR, 0x00070001) // _ADR: Address
}
Device (UHC3)
{
Name (_ADR, 0x00070002) // _ADR: Address
}
}
}
So, we can see it defines 4 devices:
EH01ataddr=0x7.7(_ADR0x00070007=slot: 7, function: 7) - A USB2.0 controllerUHC1ataddr=0x7.0(_ADR0x00070000=slot: 7, function: 0) - 1st USB1.x controllerUHC2ataddr=0x7.1(_ADR0x00070001=slot: 7, function: 1) - 2nd USB1.x controllerUHC3ataddr=0x7.2(_ADR0x00070002=slot: 7, function: 2) - 3rd USB1.x controller
In ACPI source language, the _ADR defines the address in hex.
Each word breaks down into 2 parts: slot and function (in QEMU / libvirt terminology).
As it turns out, this definition is wrong for the QEMU EHCI and UHCI devices I
passed, because they each occupied a single slot. (multifunction=on was not set).
When I tried to add those devices and set addr= parameters in QEMU using those function numbers,
it gave me this error:
qemu-system-x86_64: -device ich9-usb-uhci2,id=uhci2,bus=pcie.0,addr=0x7.0x1: PCI: single function device can't be populated in function 7.1
Evidently the ich9-usb- uhci* & ehci* devices are all "single function" devices,
unless the device at function 0x0 has the multifunction=on parameter set.
So, unless we change our QEMU device addresses to match the default SSDT, we must fix the SSDT so it matches what devices and possible PCI addresses QEMU will give the VM.
That means in our example each of those USB controllers would need to occupy a
single slot, without any sub-function numbers attached to them.
Alternatively, we could choose to set multifunction=on for the first device
and match sub-device slot numbers to the original SSDT-EHCI.aml.
In this guide, I will choose to edit the SSDT to demonstrate how it's done.
To fix this, we can:
- Edit the
SSDT-EHCI.dslfile - Recompile it with
iaslinto a newSSDT-EHCI.aml - Place it inside the
OpenCore.qcow2image in the proper directory, using the same steps we used to mount the image and partition before.
Knowing that QEMU q35 machine has certain reserved & default PCI addresses,
we can edit the QEMUUSB ACPI table to fix those PCI addresses it defines:
/*
* Intel ACPI Component Architecture
* AML/ASL+ Disassembler version 20240927 (64-bit version)
* Copyright (c) 2000 - 2023 Intel Corporation
*
* Disassembling to symbolic ASL+ operators
*
* Disassembly of /tmp/SSDT-EHCI.aml
*
* Original Table Header:
* Signature "SSDT"
* Length 0x000000A3 (163)
* Revision 0x01
* Checksum 0xBF
* OEM ID "KGP"
* OEM Table ID "QEMUUSB"
* OEM Revision 0x00000000 (0)
* Compiler ID "INTL"
* Compiler Version 0x20190509 (538510601)
*/
DefinitionBlock ("", "SSDT", 1, "KGP", "QEMUUSB", 0x00000000)
{
External (_SB_.PCI0, DeviceObj)
External (_SB_.PCI0.S38_, DeviceObj)
Scope (\_SB.PCI0)
{
Device (EH01)
{
Name (_ADR, 0x001a0000) // _ADR: Address bus=pcie.0 addr=0x1a.0
}
Scope (S38)
{
Name (_STA, Zero) // _STA: Status
}
Device (UHC1)
{
Name (_ADR, 0x00070000) // _ADR: Address bus=pcie.0 addr=0x07.0
}
Device (UHC2)
{
Name (_ADR, 0x00080000) // _ADR: Address bus=pcie.0 addr=0x8.0
}
Device (UHC3)
{
Name (_ADR, 0x00090000) // _ADR: Address bus=pcie.0 addr=0x9.0
}
}
}
In the edited version of SSDT-EHCI.dsl above, I've matched the
ehci USB 2.0 controller to QEMU's default PCI address location of 0x1a.0.
This is an arbitrary choice, but can sometimes help in the case when not
specifying a default address for QEMU CLI -device arguments.
However, it can often be best to avoid ambiguity, so declarative CLI params can
help to avoid SSDT mismatch. In the examples below, we will choose to specify
the QEMU device addresses with 'addr'. Finally, we will choose to avoid
passing multifunction=on, so all slot numbers are zero '.0'.
Next, I've changed all the addresses for uhci USB 1.1 devices to different
unused QEMU PCI slot numbers: 0x7.0, 0x8.0, and 0x9.0.
To add these devices to the VM, I passed QEMU CLI args:
## Ensure ACPI SSDT matches this
# USB2.0 EH01 in ACPI SSDT _ADR = 0x001a0000
-device 'ich9-usb-ehci1,id=ehci,bus=pcie.0,addr=0x1a.0'
# USB1.1 UHC1 in ACPI SSDT _ADR = 0x00070000
-device 'ich9-usb-uhci1,id=uhci1,bus=pcie.0,addr=0x7.0'
# USB1.1 UHC2 in ACPI SSDT _ADR = 0x00080000
-device 'ich9-usb-uhci2,id=uhci2,bus=pcie.0,addr=0x8.0'
# USB1.1 UHC3 in ACPI SSDT _ADR = 0x00090000
-device 'ich9-usb-uhci3,id=uhci3,bus=pcie.0,addr=0x9.0'
## Finally, attach the usb-kbd and usb-tablet to first USB2.0 device's EHCI bus with 'ehci.0'
-device 'usb-kbd,bus=ehci.0,id=input0,port=2'
-device 'usb-tablet,bus,ehci.0,id=input1,port=3'Translated into libvirt XML format:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<!-- ... Original VM definition ... -->
<qemu:commandline>
<!-- ... other args ... -->
<qemu:arg value='-device'/>
<qemu:arg value='ich9-usb-ehci1,id=ehci,bus=pcie.0,addr=0x1a.0'/>
<qemu:arg value='-device'/>
<qemu:arg value='ich9-usb-uhci1,id=uhci1,bus=pcie.0,addr=0x7.0'/>
<qemu:arg value='-device'/>
<qemu:arg value='ich9-usb-uhci2,id=uhci2,bus=pcie.0,addr=0x8.0'/>
<qemu:arg value='-device'/>
<qemu:arg value='ich9-usb-uhci3,id=uhci3,bus=pcie.0,addr=0x9.0'/>
<qemu:arg value='-device'/>
<qemu:arg value='usb-kbd,bus=ehci.0,id=input0,port=2'/>
<qemu:arg value='-device'/>
<qemu:arg value='usb-tablet,bus,ehci.0,id=input1,port=3'/>
</qemu:commandline>
<!-- ... rest of VM definition ... -->
</domain>
After making these changes, I recompiled the SSDT and placed it into the OpenCore.qcow2 image:
$ iasl /tmp/SSDT-EHCI.dsl
Intel ACPI Component Architecture
ASL+ Optimizing Compiler/Disassembler version 20240927
Copyright (c) 2000 - 2023 Intel Corporation
ASL Input: /tmp/SSDT-EHCI.dsl - 1322 bytes 11 keywords 0 source lines
AML Output: /tmp/SSDT-EHCI.aml - 163 bytes 0 opcodes 11 named objects
Compilation successful. 0 Errors, 0 Warnings, 0 Remarks, 0 Optimizations
$ cp /tmp/SSDT-EHCI.aml /tmp/oc/EFI/OC/ACPI/SSDT-EHCI.aml
$ sudo umount /tmp/oc
$ sudo qemu-nbd --disconnect /dev/nbd0
/dev/nbd0 disconnectedAfter booting the VM again, the QEMUUSB ACPI table is found!
The mouse and keyboard worked, and the following was output in the kernel debug console:
ACPI: SDT 0x0000000079E4000 0000A3 (v01 KGP QEMUUSB 0000000 INTL 2024097)
Now the VM has USB controllers with addresses matching the SSDT-EHCI.aml.
I hope this helps demystify some of these things.