Tutorial: Passing an Intel GPU to a Linux/KVM Virtual Machine

When running a Linux desktop it sometimes becomes necessary to run a Windows system. When GPU acceleration is necessary, dual booting often becomes the solution. But dual booting is kind of cumbersome so it would be ideal if we could get GPU acceleration in a Windows 10 VM.

Intel has a solution for this – GVT-g. From the GVT-g project website:

Intel GVT-g is a full GPU virtualization solution with mediated pass-through (VFIO mediated device framework based), starting from 5th generation Intel Core(TM) processors with Intel Graphics processors. GVT-g supports both Xen and KVM (a.k.a XenGT & a.k.a KVMGT). A virtual GPU instance is maintained for each VM, with part of performance critical resources directly assigned. The capability of running native graphics driver inside a VM, without hypervisor intervention in performance critical paths, achieves a good balance among performance, feature, and sharing capability.

After the installation we need to ensure that libvirt and qemu are new enough on your system.

$ libvirtd -V
libvirtd (libvirt) 6.1.0

Libvirt should output at a minimum version 4.6.0. If the version reported here is older than that this tutorial will not work.

$ qemu-system-x86_64 --version
QEMU emulator version 4.2.0 (qemu-4.2.0-7.fc32)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers

Qemu should output at minimum version 4.0.0. If the version reported here is older than that this tutorial will not work.

It may also be a good idea to add your own user to the libvirt and kvm groups:
sudo usermod $(id -un) -a -G kvm,libvirt
This isn’t strictly necessary but it makes using virt-manager easier. Without doing this you will have to type your password every time you want to start a VM.

Configuring the host to support GVT-g

In order to support GVT-g we need to add a parameter to the boot loader, add several modules to be automatically loaded at boot, and create a simple systemd-unit to automatically create the virtualized GPU at boot. We will also need to reboot the system; while you could reboot the system after each step, when following this guide only one reboot should be necessary.

Configuring the bootloader

Open /etc/default/grub using your favorite text editor and find the line that starts with GRUB_CMDLINE_LINUX and add i915.enable_gvt=1 i915.enable_fbc=0 at the end of the line. On my system the file looks like this after the edit:

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="gfxterm"
GRUB_CMDLINE_LINUX="resume=/dev/mapper/vg_isla-swap rd.luks.uuid=luks-123568b4-82fb-4cb0-aeed-7cfd3c2603f2 rd.md.uuid=8d843e1d:c4eda4f4:3622e122:9ef0c927 rd.lvm.lv=vg_isla/root rd.md.uuid=f978b92e:db7efe36:2c7037ed:2a6a6dcf rd.lvm.lv=vg_isla/swap rhgb quiet i915.enable_gvt=1 i915.enable_fbc=0 amdgpu.dc=0"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
GRUB_FONT="/boot/grub2/fonts/DejaVuSansMono.pf2"

After making the change we need to regenerate the grub2.cfg file.
On Fedora with UEFI boot: sudo grub2-mkconfig -o /etc/grub2-efi.cfg
On Fedora with BIOS boot: sudo grub2-mkconfig -o /etc/grub2.cfg
On Ubuntu: sudo update-grub

If you don’t know whether you are BIOS booted or UEFI booted see if the directory/sys/firmware/efi/efivars/ exists. If it does not, you are BIOS booted.

If your system does not use Grub please refer to your distribution’s documentation on how to change kernel parameters.

Loading the GVT-g modules at boot

Use your favorite text editor to create a file called /etc/modules-load.d/kvm-gvt-g.conf and place the following content in it:

kvmgt
vfio-iommu-type1
vfio-mdev

At this point you have to reboot your host machine.

Checking whether GVT-G is working

We first need to find out what the PCI address of our Intel GPU is. The easiest way to do this is to run the lspci tool and note the PCI address. On my system the output looks like this:

lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 630 (Mobile) (rev 02)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon Pro WX 7100 Mobile]

In this case the Intel GPU’s PCI address is 00:02.0. Keep a note of it as we will need this address several times in the future. We now need to go to the device directory of the GPU in /sys and check whether GVT-G is enabled.

cd /sys/bus/pci/devices/0000\:00:02.0

There should be a directory called mdev_supported_types in this device entry. If there is not, check /proc/cmdline and see if the bootloader changes were applied.

Creating the virtual GPU

Creating a virtual GPU works by putting a UUID value into the create virtual file of a virtual GPU type. We first need to check what GPU sizes your GPU supports (assuming you are still in the GPU’s device directory in /sys):

$ ls -l mdev_supported_types/
total 0
drwxr-xr-x. 3 root root 0 May 15 03:32 i915-GVTg_V5_4
drwxr-xr-x. 3 root root 0 May 15 03:32 i915-GVTg_V5_8

This means that my system supports two sizes of virtual GPU V5_4 and V5_8. The size determines how much video RAM the virtual GPU will have. I have tested the two models in my system and I couldn’t find any performance difference so I’m going to continue assuming it doesn’t matter for this post and create the smallest vGPU. Note the larger the numbers the smaller the vGPUs. In this case V5_8 being the smallest.

Each vGPU needs a UUID. The easiest way to generate a UUID is by running the uuidgen tool. For instance:

$ uuidgen
cb33ec6d-ad44-4702-b80f-c176f56afea1

Keep a note of your UUID or just use the one here. It doesn’t matter that much if you only need one vGPU.

echo cb33ec6d-ad44-4702-b80f-c176f56afea1 | sudo tee mdev_supported_types/i915-GVTg_V5_8/create

If you get an error like No space left on device you may not have enough vRAM assigned to your Intel GPU in your firmware. You will need to go into your system’s configuration utility and raise it. You will need to refer to your system’s documentation on how to do this. Firmware configuration utilities vary wildly per vendor but in general you will be able to enter it by pressing F2 or DEL during early boot.

If the command succeeded you should now have a directory named after the UUID in the GPU’s /sys directory.

$ ls
ari_enabled                           class                     current_link_width  driver           firmware_node  i2c-3  iommu        link            max_link_width        msi_irqs   rescan     resource2     rom               uevent
boot_vga                              config                    d3cold_allowed      driver_override  graphics       i2c-4  iommu_group  local_cpulist   mdev_supported_types  numa_node  reset      resource2_wc  subsystem         vendor
broken_parity_status                  consistent_dma_mask_bits  device              drm              gvt_firmware   i2c-5  irq          local_cpus      modalias              power      resource   resource4     subsystem_device
cb33ec6d-ad44-4702-b80f-c176f56afea1  current_link_speed        dma_mask_bits       enable           i2c-2          index  label        max_link_speed  msi_bus               remove     resource0  revision      subsystem_vendor

Persist the vGPU creation at boot

Creating the GVT-G device is not permanent, at the next reboot the vGPU just created will need to be recreated. We can create a systemd unit file to automate this process during boot.

Create a new file called /etc/systemd/system/setup-gvt.service and place the following content in it, replacing the UUID, PCI address, and vGPU size with the values we found for your system:

[Unit]
Description=Setup GVT

[Service]
Type=oneshot
ExecStart=/usr/bin/bash -c 'echo cb33ec6d-ad44-4702-b80f-c176f56afea1 > /sys/devices/pci0000:00/0000:00:02.0/mdev_supported_types/i915-GVTg_V5_8/create'

[Install]
WantedBy=multi-user.target

Make sure it starts automatically at boot with sudo systemctl enable setup-gvt

Installing the Windows 10 VM with Virtio hardware

Next I will provide a step-by-step guide with screenshots below. If you’re already familiar with virt-manager and Windows 10 setup these are the highlights:

• Create a Qemu/KVM instance (Not a user session)
• Select the Windows 10 .iso as installation media
• Make sure to select ‘Customize configuration before install’
• Set the CPU type to host-passthrough
• Set the hard drive type to VirtIO
• Set the network controller type to VirtIO
• Get to the drive selection screen
• Swap the Windows 10 iso with the virtio windows driver iso.
• Select the NETKVM/w10/amd64 directory and install the network controller driver.
• Select the viostor/w10/amd64 directory and install the virtio SCSI driver.
• Swap the Windows 10 iso back and continue the installation like normal
• Install the Windows guest addons from https://spice-space.org

Press the 'create a new virtual machine' button in virt-manager

Ensure that QEMU/KVM is selected and not a user session

Select the Windows 10 iso downloaded earlier

Create a harddrive for the VM

Select a name and be sure to select 'Customize configuration'

Ensure the firmware is set to 'BIOS'

Set the cpu type to 'host-passthrough' and set a sane cpu topology.

Set the hard drive type to VirtIO

Set the network hardware type to VirtIO

And click 'begin installation'

Windows setup will start, press next

and press install now

Type your Windows product code

Accept the license

Select custom

At this point there won't be a hard drive to install to. We will install the drivers from the VirtIO iso we downloaded earlier. Click 'load driver'

This screen should pop up. Go to the virt-manager interface and 'view' select 'details'

Now select the VirtIO driver iso, hit apply, and select 'view' 'console'

Select the cdrom drive

Select the NETKVM/w10/amd64 directory and click OK

Install the Network card driver

At this point we still won't have a hard drive. Load another driver

This time select the viostor/w10/amd64 directory and click OK

Load the driver

Go back to details, and select the windows 10 iso agian, click apply and go back to 'console'

Now we will have a drive, click 'New'

Create the partitions

Press 'OK'

Select the new partition and click next

Windows install will commence

Follow the whole setup process

eventually...

There we go

Go to 'spice-space.org'

Get to the download page and download the windows drivers

Run the setup, acknowledge the prompt

Next

Make sure to click 'install'

And we're done

Shut down the VM

At this point we have a relatively standard Windows 10 installation with VirtIO drivers. We should make a backup of the libvirt configuration now as we will be making changes to it.

sudo virsh dumpxml win10 > win10.xml

Adding the vGPU to the Windows 10 VM

For this step we will have to edit the libvirt XML file directly as virt-manager does not support the device types we will need to add. We will be making the following changes:

• Setting the ‘domain’ type to QEMU
• Configuring the Spice server to use GL and disable networking
• Removing the virtualized VGA adapter
• Add the vGPU mediated PCI device
• Enable RAMFB for early boot messages
• Enable DMABUF so we can see accelerated GPU output in our virt-manager window
• Enable a workaround for Mesa bug #2678

First, we will make a backup of the win10.xml file we created earlier. cp win10.xml win10.xml.bak in case we make an error editing the instance. For the rest of this section we’re assuming you have the win10.xml file open in a text editor.

Setting the ‘domain’ type to QEMU

To set the domain type to qemu we need to edit the first line of the file. Change the line:

<domain type='kvm'>

to

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

Configuring the Spice server to use GL and disable networking

Now to change the Spice client settings. Find the graphics block, it should look something like this:

<graphics type='spice' autoport='yes'>
  <listen type='address'/>
  <image compression='off'/>
</graphics>

Change it to look like this:

<graphics type='spice'>
  <listen type='none'/>
  <gl enable='yes' rendernode='/dev/dri/by-path/pci-0000:00:02.0-render'/>
</graphics>

Note that we need to put the PCI address of our real Intel GPU in the rendernode of this block.

Removing the virtualized VGA adapter

We now need to disable the QXL virtual VGA adapter but we can’t just remove it as libvirt seems to require at least one video block. Find the video block, it should look something like this:

<video>
  <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</video>

Change it to look like this:

<video>
  <model type='none'/>
</video>

Add the vGPU mediated PCI device

Now to add the vGPU. We need to add a whole new block. You can place this block directly below the video block we just edited.

<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='on'>
  <source>
    <address uuid='cb33ec6d-ad44-4702-b80f-c176f56afea1'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</hostdev>

Enable RAMFB for early boot messages

Finally, we need to enable DMABUF and RAMFB. For this we need to add another entirely new section, but this time it needs to go at the very end of the file right before the closing </domain> line. The following block needs to be put directly above the last line of the file:

<qemu:commandline>
  <qemu:arg value='-set'/>
  <qemu:arg value='device.hostdev0.x-igd-opregion=on'/>
  <qemu:arg value='-set'/>
  <qemu:arg value='device.hostdev0.ramfb=on'/>
  <qemu:arg value='-set'/>
  <qemu:arg value='device.hostdev0.driver=vfio-pci-nohotplug'/>
  <qemu:env name='INTEL_DEBUG' value='norbc'/>
</qemu:commandline>

At this point our XML file is ready and we can load it back into libvirt by running sudo virsh define win10.xml. At this point it should be possible to boot the VM again.

After making the above edits your win10.xml file should look something like this:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>win10</name>
  <uuid>1b7fa3cc-b3ba-464b-8585-1423514c1067</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' dies='1' cores='2' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/win10.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/hp/Downloads/Win10_1909_English_x64.iso'/>
      <target dev='sdb' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:d4:70:45'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice'>
      <listen type='none'/>
      <gl enable='yes' rendernode='/dev/dri/by-path/pci-0000:00:02.0-render'/>
    </graphics>
    <sound model='ich9'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
    </sound>
    <video>
      <model type='none'/>
    </video>
    <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='on'>
      <source>
        <address uuid='cb33ec6d-ad44-4702-b80f-c176f56afea1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.hostdev0.x-igd-opregion=on'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.hostdev0.ramfb=on'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.hostdev0.driver=vfio-pci-nohotplug'/>
    <qemu:env name='INTEL_DEBUG' value='norbc'/>
  </qemu:commandline>
</domain>

Configuring the VM

After booting the VM you will notice that the VM is very slow and running in a small window. You can still login to it by hitting enter and typing your credentials. At this point this just let it sit for several minutes. Windows will download the appropriate Intel GPU drivers and install them. The screen will go black for a little while and then the VM should return with a normal speed, but still running in a small window. The results will be something like this:

We will fix the resolution issue in a minute, but first we should install the full-fat Intel GPU drivers. To make the VM a bit easier to use you can go to ‘view’/’scale display’ and select ‘always’ in the virt-manager ui. This should result in something like this:

To install the Intel display drivers we need to do the following steps:

• Open a browser and go to https://downloadcenter.intel.com
• Scroll down and select ‘graphics’
• Download the DCH drivers .exe
• Run the installer and reboot

Go to downloadcenter.intel.com

Scroll down to 'graphics'

Select the DCM drivers

Scroll down and select the .exe installer

Press run

Allow the installation

Next, next, next. This is where the installer will pause for a bit as it is installing the actual driver

Select reboot and hit finish

Right now the resolutions that your VM supports is pretty limited. Right-click on the desktop and select ‘display settings’.

We can fix this with a tool called ‘custom resolution utility‘:

• Download ‘custom resolution utility’ from monitortests.com
• Unzip to a directory, for instance C:\CRU
• Run CRU and add the simple resolutions you want
• Run restart64.exe from CRU
• Set the new resolution in ‘display settings’

Search for 'custom resolution utility'

Download the zip file and open it

copy the contents

create a new directory like C:CRU

Paste, and run CRU

CRU needs admin permissions

Press 'Add' in the 'Standard resolutions' box and add the resolutions you want

Press 'OK' once you're done

Run 'restart64.exe'

Restart64 also needs admin permissions

Exit the tool

Going back to 'display settings' should now let you select different resolutions.

Validating the installation

At this point everything should work. To validate this we will run dxdiag.

• type dixdag in the bottom left of the task bar
• Run dxdiag
• Check to see that DirectX 12 is supported

Run dxdiag

Validate that you have DirectX12

Congratulations! You’re done! You should now have a Windows 10 VM with GPU acceleration. It should be possible to run games that work on the iGPU, run Godot engine, or other 3D workloads.

If you have any comments or issues please feel free to leave a comment below. You can sign in with your existing Google, Twitter, or Github accounts!