Vfio Versions Save

Its been a while since the last release tag and with bridging revamped in master its time.

As usual there's been many various script handling improvements, cleanup, documentation tweaks, clarity in printed messages and additional/combined argument flags for ease of use. Below are the more noteworthy changes.

Key changes since 0.7.1:

• Revamped advanced networking arguments
  • -bridge tap0,br0,eth0 is now expressed as -bridge br0, -tap tap0, -hostint eth0.
  • The script still reacts to these arguments depending on how many are specified.
  • That is to say it will still create bridges automatically if all three arguments are provided and can attach the VM to a new or existing tap which may already be configured.
  • Or even just a tap alone with no bridges for host configuration or external software for handling the guest's tap interface.
• Win11 support with the -tpm argument to spin up a software tpm (Using swtpm)
• By default (-pinvcpus not specified) the guest now takes up only half of the host's total CPU thread count
  • This has been done because in certain circumstances a Windows 10/11 VM may 100% all cvpu's for a prolonged period of time. As a result of this insane full package load the host can sometimes show signs of locking up as this script gives the guest scheduling priority.
  • Noticeable times Windows will full throttle all its given vcpus include when the bootloader fails to boot and when shutting down with fastboot enabled.
• Improved automated troubleshooting output for validating the kvm module is ready for use.
• Add -force to ignore the script's error reporting (Useful in some testing cases)
• Add -noBios/-legacy/-legacyboot` for booting without OVMF (For running MBR/BIOS VMs and P2V's)
• Skip the compact_memory call for ZFS hosts running kernel 6.3.1+ to avoid https://github.com/openzfs/zfs/issues/15140
• Add -nvme to present the VM's disk to it using the NVMe protocol
  • Most OSes pack only the drivers they need into their bootloaders so if they were on NVMe then giving them a VirtIO disk will be missed by the bootloader.
  • This has been added primarily for when I try to boot a physical disk in a VM and the OS only has NVMe drivers in its bootloader.
  • It has also proven helpful in P2V and V2V scenarios where the same problem exists.
• -hostAudio quality of life improvements
  • Automatic detection of Pulseaudio or Pipewire for selecting the best -audiodev for QEMU to use
  • Sanity checking for whether QEMU has the required audio libraries of each to proceed
  • Fallback to pulseaudio if pipewire is selected but pipewire-pulse isn't installed.

For the full changelog experience with all the small bits heres the commits between 0.7.1 and 8.0 https://github.com/ipaqmaster/vfio/compare/0.7.1...0.8

A small bump for some new features.

Key changes since 0.7:

• Various script subroutine cleanup to cover all use cases even when certain requirements aren't configured on a host

  • e.g. Checking only for IOMMU-related stuff when the script is about to engage it

• Further documentation cleanup as usual

• Added -iommu as an alternative IOMMU group check flag. Still keeping -iommugroups and -iommugrouping

• Added -portforward to port-forward host ports to the guest when NOT using a bridge. This is for the User-mode guest networking default setting (NAT). Pretty useful for tests and quick start/stops.

• Added Multi-GPU VFIO support without having to kill an existing X server.

  • This is supported in scenarios where Xorg is configured via xorg.conf to ignore your other host GPUs. The README.md contains example xorg configuration.
  • This feature allows for a GPU to participate in CUDA operations for the day while also being able to take an unbind request on the fly (Obviously not while something is in active use of the card).
  • The script consults fuser to look for any processes (Or Xorg itself) using the GPU if any before unbinding.
  • If Xorg isn't using the GPU but user processes are; it kills them and starts them back up again after getting a target card onto vfio-pci.
    • This has been useful for me as Firefox, Chromium based browsers, XFCE4 and Cinnamon (Any Gnome based WM, really) continue to start processes which latch onto my second card even after asking xorg.conf not to.

• Added a new flag -killX to tell the script that killing X is OK. This was primarily added for Single GPU Passthrough scenarios.

• Fixed a niche issue with hugepage allocation in special circumstances.

Other than the usual cleanup some new flags have been aded which I found useful enough to make permanent:

1. Flag(s) [-avoidVirtio/-noVirtio] to use generic virtual hardware such as e1000e for guest Ethernet among other comprompises for special cases.

   • If you don't have the virtio ISO handy during installation or are booting a kernel/initrd directly without virtio drivers baked in, this is a savior.

2. Flag(s) [-cputhreads / -showpairs / -showthreads / -showcpu]

   • To quickly show which host threads belong to what cores.

3. Flag(s) [-ignorevtcon / -ignoreframebuffer / -leavefb / -leaveframebuffer / -leavevtcon] Optional workaround for kernel Bug 216475 on Nvidia cards.

   • At the moment, unbinding and rebinding vtcon0/1 may cause a null pointer dereference in fbcon.c with certain video drivers. True for kernel version 6.0.11 at least.
   • With this argument I can resume passing my GPU from guest to host indefinitely in a single boot but at the cost of my virtual consoles.
   • I like my virtual consoles though so hopefully this is fixed soon and this flag gets deprecated.

4. Added -kernel, -cmdline and -initrd options for various realtime kernel hackery and testing

As for other general changes since the last tag:

• Squashed vcpu pinning bug introduced in earlier commit (Had the ampersand in the wrong spot, damn)

• -m can now also be used instead of -memory

• Script no longer relies on brctl, bash now traverses /sys/class/net itself.

• Hugepage arguments now also get calculated in the dry run

• Script automatically increments its spice port if already in use

Script has been stable to a point where it's worth tagging a new checkpoint given how long it's been since the last one.

Some noteworthy things since 0.6a which was quite a while ago:

• Various script printing changes and stability improvements for certain run cases. (This sentence covers most commits)
• Script uses the -nodefaults qemu argument to avoid undesired default VM hardware the easy way.
• Script uses the qemu serial device on VM start which can be multiplexed into the qemu terminal for a guest serial interface or live VM management and hardware attaching via the qemu command line if needed.
• Script uses a backwards compatible qemu-xhci USB controller on the guest to support USB3 devices.
• -hostaudio argument added to start a pulseaudio server on the host if possible and not already running while attaching a ich9-intel-hda sound device to the guest.
• -quiet flag added with some printing changes to make the script as quiet as possible unless a fault occurs. Something I found more useful after a few hundred runs.
• Guest network MAC address generation improvements and using openssl to generate the random MAC if ever required.
• Bridger function changed to expect the tap interface as its first argument (e.g. -bridge tap5,br1,eth0), VM can now be started with only a tap adapter and no bridge for host-only networking
• Looking glass qemu argument format changed to the classic format to support qemu 4.2.1 which is shipped on Ubuntu.

Some taggable changes since 0.6:

• Each virtual disk passed through using the -image argument gets an iothread each.

• guest vcpu threads of qemu get pinned to each specified core using -pinvcpus (replaces -taskset) then chrt is used to set all vcpu qemu threads to use the FIFO scheduler. Script waits dynamically for vcpu threads to spin up as startup time varies host to host and depends on what's being passed through.

• -nodefaults added to qemu arguments by default. VM networking and display managed solely by script.

• logic for hugepages improved with support for specifying a custom mountpoint.

• Network bridging cleaned up with better sanity checking and custom guest mac addressing to prevent conflicts (Say if more than one VM is running with this script).

• Automatic color toggling, useful for piping/grepping output without having to manually include -nocolor.

• Various other small bug fixes and script tidy-up.

Enough changes since 0.5b with stability on two machines to tag a release for it.

Fluff: Fired up previous desktop (Sabertooth x79, 3930K GTX660 [host] GTX780[Guest] to test some dual GPU support scenarios. Not the most responsive environment however useful for testing the script.

Some changes in 0.6 since 0.5b:

1. "Script cleanup" as usual this time featuring better echoing, some improved function return logic, catches for driver unbind hangs and other redundancies to exit early if it thinks something went wrong.

2. Better detection for a GPU device and the script will only attempt to kill a display-manager service if the GPU device is found currently bound to anything other than vfio-pci, rather than always. The kill is only kept in the script to try and avoid a possible deadlock during gpu unbinding.

3. Added a -lookingglass/-looking-glass/-lg flag to support the Looking Glass project in this script. Adds some shared memory and spice devices to qemu when specified.

   • Anyone trying to use this flag will still need to install the looking glass host software in their VM and add the VirtIO IVSHMEM driver to the "PCI standard RAM Controller" which appears in Device Manager under System Devices before Looking Glass will function.

4. Added a -romfile/-vbios added which will be shoehorned into the qemu pci argument for any GPU device on detection

5. -hugepages/-huge improved to detect the system's hugePageSize better and support optional custom mountpoints when specified after the argument. Otherwise falls back to default /dev/hugepages, but will no longer automount it (though, it should be automounted on modern distros anyway). On top of all this, if it detects hugepages are already allocated AND there's enough for how much memory the VM wants, it will use the existing free pages instead of allocating freshly. Otherwise, it will drop caches, compact memory and allocate its own as usual.

   • Made this change to support 1GB hugepages which may not always be mounted at /dev/hugepages and may have its own mount options modifying the page size, which the script should detect dynamically.

   • With this I was able to add hugepagesz=1G hugepages=16 to my kernel boot parameters to sactifice 16GB at system startup to hugepages, and mount those pages with mount -t hugetlbfs hugetlbfs -o rw,relatime,pagesize=1G /dev/1gb_hugepages and start my VM with -hugepages /dev/1gb_hugepages included to use the pages which are already present. Even if not preallocated the script would be capable of allocating 1GB hugepages instead of 2MB sized ones on the fly worst case.

Has been stable in my day to day usage, might as well tag it as there's some changes worth having if someone downloads a release instead of pulling master.

Overall since 0.5a:

1. Mostly clean up of readability and some functions such as hugepage allocation, display-manger management and other minor places.

2. Switched to using tee with sudo instead of shell redirection to deal with the script possibly running as non-root.

3. -hyperv flag added to add some of the most common hyperv enlightenments

4. Permissions management function added to take ownership of hugepages, disks, virtual disks, vfio groups, the bridge tap interface and otherwise before qemu starts when the script isn't running as root, then return them during cleanup. This were added as the script has been changed to not expect root privileges instead running everything as my user but opening gates where required with sudo before qemu gets started. Modern qemu versions can drop permissions after starting so I may look into that later.

Minor script changes which might as well be tagged.

1. Arguments are now case-insensitive because that's annoying.
2. Script color can now be disabled with -nocolor or -nocolour (optional s on the end)
3. Fixed a mistake where the "return color" was white, but not "none", overriding a terminals color to white after running the script when it may have been something else. If your terminal foreground color was white like mine you may not have noticed this either.

Worth tagging a new one as there's been a few changes to the script since 0.4c and not everybody likes to pull master and pray.

Mostly quality-of-life changes, bugfixes with a few actual features. Noteworthy changes since 0.4c as below:

• Cleaned up the script a little. More consistency in key parts and reuse of common logic checks.

• Colorized most of the scripts own output so you'll see juicy orange and red coloring if it has something important to share, otherwise green or just default. (Also comes with -colortest if someone wants to check it works)

• Added -hugepages to support allocating hugepages before starting qemu, and giving it a relevant memory argument. One will still need to customize their hugepagesize manually (default seems to be 2MB) but this call will read to make sure it allocates only as many as required if a non-default value.

• Network bridging now supports attaching to an existing bridge and cleaning up only the tap adapter when finished

• Made the -image flag optional, allowing liveCD usage without any trouble.

• Added -imageformat to specify the format of the image separately (Or not at all, qemu will complain though)

• Added -nonet for when you don't want any virtual networking adapter to be passed to the VM. Useful if you're about to use -PCI to hand it a network card of its own.

• The enumeratePCI function now only kills the display-manager and unbinds the EFI console driver if it detects a specified GPU is hooked by a driver. But everything else is free game. This has been useful for passing things to a VM without bothering the X session such as USB Controllers, Network/Sound cards, or maybe a *different GPU who's is already unbound? (dual-gpu setup yet to be tested)

• Added -iommugroups to print IOMMU grouping of PCI devices in a list if possible, then exit.

Fixed up the consoleDrivers function to prevent MMAP issues when starting a VM with a GPU passed through along with the fixes mentioned in pre-release 0.4b.

Tested working on the Aorus Pro x570