Short Version

I followed the excellent PCI passthrough via OVMF Arch Linux wiki article and everything works well.

Long Version

Motivation

Throughout the years, I’ve had a dedicated Windows computer sitting around. It would typically sit idle 99% of the time until I needed to build/test Windows software, update the firmware of some device, or decided to play some Windows exclusive game.

Even though I’ve known that passing through PCI devices has been successful for many people for a few years, my first try at enabling IOMMU support on an AMD machine caused an early Linux boot up failure which took enabling various eccentric boot flags to diagnose. However, I’ve seen a lot of recent success stories so I figured the stack has matured to the point that it was worth another attempt.

Host OS

I’ve been using Arch Linux as my main OS for years so not much was needed to do here but install the required packages and add my user account to the appropriate groups.

Base Hardware

• CPU: Intel Core i7-4790K
• RAM: Crucial 32GB DDR3-1600
• Motherboard: Asus H97-PRO GAMER
• Primary/Host GPU: AMD Radeon RX 480
• Secondary/Guest GPU: NVIDIA GTX 1080

Step 1 - Prerequisites

I enabled intel_iommu=on iommu=pt but I soon realized having “Intel Virtual Technology” (VT-x) enabled in the BIOS was not enough as there was only a single line of output relating to the IOMMU in the dmesg. After some digging, I was able to locate a separate option in the BIOS to enable VT-d. I then checked the IOMMU groups and they were valid/what I was hoping they would be. At this point I figured I was all set to begin in earnest. I downloaded the Windows 10 Fall Creators Update ISO image (Win10_1709_English_x64.iso) direct from Microsoft.

Step 2 - Test Install of Windows 10

I decided to create and install a new Windows 10 virtual machine without passing any devices through to make sure that just a basic VM worked. I used virt-manager to create the machine and disk image and was able to complete an install without any problems. After doing some basic testing, everything seemed to work fine using the virtual audio/display adapters and I was confident enough to try passing through some physical devices.

Step 3 - Real Install of Windows 10

I pulled out the SSD from my Windows machine and tried to boot off of it but I was unable to do so which I sort of expected. The VFIO Windows drivers weren’t loaded and I was unable to figure out how to do so using the Windows recovery/repair tools. I decided to re-install Windows 10 over top of my existing install and let it move my data to Windows.old. This ended up working out well since I was planning on removing a bunch of software on there anyway. I deleted a bunch of folders in Windows.old that I no longer needed or cared about and then shuffled Steam/GOG games back into their old locations and used the Steam/GOG clients’ verify/repair functionality to get the games working again.

Step 4 - Passing through more devices

Since passing through the SSD worked fine, I moved on to passing through the NVIDIA GPU and a single USB keyboard/mouse combo device. Passing through these devices worked OK but there were some performance/ease of use problems which I knew I would have to deal with shortly.

Step 5 - Performance Tuning

I had to follow a few steps from the performance tuning/troubleshooting sections of the aforementioned wiki article, namely:

• CPU pinning - I pinned the VM to the hyper-threaded cores of my CPU
• Static huge pages - I’m using 12 1GB pages to reduce page table walks/TLB thrashing
• Enable Message Signaled-Based Interrupts - I edited the Windows registry to enable MSI on both devices exposed by the GPU to fix HDMI audio problems

With those items complete, all performance concerns were addressed.

Step 6 - More hardware to increase ease of use

At this point I was pretty happy but there were two dangling problems:

1. Network issues
   • The default of NAT didn’t work for me since I wanted to talk directly to other machines on the network.
   • Performance was rather inconsistent/bursty.
   • I considered a bridge but I was worried about conflicts with some other VMs/bridges I have planned.
2. USB issues
   • My machine reports 4 USB controllers but 3 seem to be unwired/phantom because all 12 USB 2.0/3.0 ports are connected to the same controller.
   • I have two Logitech Unifying Receivers and despite being different revisions they have the same product/device IDs so depending on boot ordering, the wrong one will get passed through to the VM.
   • I wanted to pass through a whole controller so I could hotplug devices and have Windows see them but this wasn’t possible.

So I opted to add a PCIe ethernet adapter and a PCIe USB 3 controller. Thankfully they were also put into isolated IOMMU groups so I was able to pass them through without incident. I also added a USB 3 hub so I could hotplug USB devices easily.

After a week of use

I briefly tested a few games like GTA V, Witcher 3, and didn’t experience any performance or compatibility problems. I settled on playing Rise of the Tomb Raider and recently completed the main game. In the ~20 hours of game time I never had any stability or performance problems. That is actually pretty amazing considering the GPU I was using was in a PCIe 2.0 slot running at 2x which limits the throughput to 1.0 GB/s (compared to 15.8 GB/s if I were using the PCIe 3.0 slot @ 16x). Digging around, I found this TechPowerUp – NVIDIA GeForce GTX 1080 PCI-Express Scaling article which had the following benchmark:

My gaming monitor is a 75Hz 1080p screen connected via HDMI and I played with VSync enabled so frames were capped to a level where I wouldn’t see performance problems. (PCIe 2.0 @ 2x = PCIe 1.0 @ 4x)

Future

It’s not ideal to have a bottlenecked GPU but it’s not noticeable for light gaming from time to time, so I’m pretty happy with my setup for now. I suppose it’s to be expected that I can play console games with reasonable performance that were “up-ported” to PC 2-3 years ago on half of 8 thread CPU that has aged pretty gracefully due to slowing of Moore’s law.

I’ll be attempting an AMD-based Windows 10/PCI passthrough setup for my daughter soon so at the very least, PCIe bandwidth concerns are something to keep in mind. I also have some Z97 Intel motherboards available to experiment with if I want to try to increase performance. There’s also the possibility of upgrading to more recent hardware but current DDR4 prices make it difficult to justify the cost to performance increase ratio.