The next level in Desktop virtualization: multi GPU pass-through
Written by Thomas Koetzing at Tuesday, 16 November 2010 | Article editor:
XenServer 5.6 multi GPU pass-through for XenDesktop HDX 3D Pro Graphics (CTX125574) is an official Citrix article (temporarily not available). What is behind the article and what does it mean for the future of Desktop virtualization that is described in the technical PDF document? I got "hands-on" with the Citrix technical preview of multi GPU pass-through and I will describe details in this article.
The next level in Desktop virtualization is the pass-through of graphics processing units (GPU) which means direct access to the graphic hardware from a virtual Machine (VM). Of cause you would need multi GPU pass-through on a Hypervisor to make it useful for not just one VM.
All vendors are working on such a solution like Microsoft RemoteFX (calista) that comes with Service Pack 1 for Server 2008 R2 Hyper-V. RemoteFX usage in the coming release will be dedicated to LAN environment and to support multimedia / areo in a much better way than before. Citrix will leverage / support RemoteFX in the future but of cause Citrix has it's own HDX solution that does works over WAN as well.
I did configure Citrix's technical preview in XenServer 5.6 RTM for a customer to use it with Citrix HDX 3D Pro Graphics (multi GPU not supported for production). The target here is to support high-end 3D professional graphics applications with mutli GPU pass-through. Applications like CAD/CAM are often using OpenGL and or Direct-X and Citrix provides the full user experience with HDX 3D Pro over ICA from everywhere. XenDesktop Enterprise version was used to give user access to the high-end 3D graphic application.
Virtual Desktops without GPU pass-through Microsoft Windows 7 installed on a Hypervisior will have the aero theme disabled, basically because the VM graphic card performance is too bad to support aero. You would need to install the virtual Desktop for instance on a blade PC, something with real hardware access to graphic cards or CPU's with multiple cores. On those Desktop you would install the Virtual Desktop Agent (VDA) from Citrix and on top HDX 3D Pro Graphics. Then you have a high graphic performance virtual Desktop as part of XenDesktop. Now who wants to manage those entire blade PC's?
Virtual Desktops with multi GPU pass-through What is required to get it to work? You start to install XenServer 5.6 on a supported Hardware, nothing special here. Next you need a graphic card with multi GPU’s and supported by the technical preview (basically Citrix means its not tested in all depth and with every graphic hardware). Supported are NVIDIA Quatro FX cards (I used FX4800) with CUDA 2.1 or above. After building in the NVIDIA card and enabling "VT for direct I/O" in the BIOS, you have to do some
changes to the XenServer to enable direct hardware access. Here it comes in a nutshell:
Add: iommu=1 iommu_inclusive_mapping=1 in /boot/extlinux.conf Enables the access to PIC devices
Execute #extlinux /boot Regenerates the bootloader
Execute #lspci to view available PCI devices and note the ID (example 07:00.0) Find the graphic device ID
Execute #xe vm-list or enable the hidden objects in XenCenter to find the VM uuid Find the uuid of the VM you want to have direct graphic access
Execute #xe vm-param-set other-config:pci=0/0000:07:00.0 uuid=uuid_from_the_vm Attach the PCI device to the VM
Important to know is that the VM will now access the Hardware and any attached screen (resolution limit) to it. Therefore my advice: The NVIDIA card should be used as the secondary card of the host. The result is that the screen in the XenCenter console of the VM looks like to hang (Windows starting...) but in fact the VM is already running! Use Remote Desktop or VNC to get access to the VM, since the console in XenCenter will not react on any input. When you have logged it to the virtual Machine the device manager will pop-up and require the NVIDIA graphic card drivers for the OS.
Next you will install the Citrix VDA which needs to be installed from the command line to enable WDDM msiexec /I XdsAgent.msi INSTALLONWDDM=1. Last step is to install Citrix HDX 3D Pro and the corresponding ICA Client for HDX 3D Pro. That's it!
Performance with GPU pass-through Guess you would try GPU pass-through and HDX 3D Pro Graphics with Windows 7 x64 and I did too. As to the HDX 3D Pro Guide a quad core CPU is required for GPU pass-through. The Windows 7 VM was configured with two vCPU (I explain later why just two vCPU) and 8GB RAM. After logging into the Machine I noticed right away the sluggish behavior, so I launched task manager. The process "picaOverlay.exe" was using almost one complete CPU constantly and is for Windows Aero! "picaDispMgr.exe" was also using the second CPU up to 100% when idle but goes down once the user starts working. So with two vCPU almost all CPU performance was used by two Citrix proccessies and that's why Citrix recommends at least a quad core CPU. Using Windows 7 basic mode instead of aero frees almost one CPU but still not very useful performance left for a CAD/CAM application!
I was discussing the CPU usage with Citrix and of course they said, we recommend more than two CPU's ("quad core") for the VM using GPU pass-through and HDX 3D Pro Graphics. This is exactly the point! Citrix is talking about quad core but what is a vCPU!? Before I answer that question let's have a look what the Client operating system Windows 7 and support on CPU's
Windows 7 supports max. two physical processors but unlimited core processors.
That means: 1 Intel Quad = 1 CPU with 4 Cores = 1 Socket = 4 Cores in Home Premium 2 Intel Quad = 2 CPU with 4 Cores = 2 Socket = 4 Cores in Home Premium 2 Intel Quad = 2 CPU with 4 Cores = 2 Socket = 8 Cores in Professional/Ultimate
The answer, a vCPU is a physical processor to the operating system without any cores! That means Windows 7 can have a maximum of two vCPU's! What current Hypervisor would need is vCPU's with cores but none of the vendors officially have that feature yet! But wait! Citrix XenServer 5.6 has a hidden feature to do exactly that, enable cores for vCPU's! The VM that should get vCPU cores needs to be shut down in order to make the changes. On the XenServer CLI use the following two commands:
#xe vm-param-set platform:cores-per-socket=n uuid=<vm-uuid> #xe vm-param-set VCPUs-max=<amount of CPUs> VCPUs-at-startup=<amount of CPUs> uuid=<vm-uuid>
After executing those command the VM with Client OS comes up and shows more than two CPU's and made my customer very happy! The Desktop virtual Machines have now one or two vCPU's with 4 cores.
Another valid point is the bandwith next to CPU and GPU. In a LAN environment no problem but over WAN it's a different story. Citrix recommends at least 1.5 MBit/s when using HDX 3D Pro Graphics. That's were Citrix Branch Repeater that works with Access Gateway helps optimizing the data traffic.
Summary The XenServer multi GPO pass-through option and HDX 3D Pro Graphics is a powerful combination to create effectively centralized high graphic performance virtual Desktops. With Citrix Branch Repeater and ICA those Desktops are accessible from everywhere with great user experience, at least my customer is happy with it.
If Citrix would come out with XenServer 5.7 that has multi GPU pass-through build in and vCPU's with cores as XenCenter option, HDX 3D Pro Graphics proccessies optimized as well as multi monitor support, then Citrix would make a next step into the future of Desktop virtualization!
What do you think about multi GPU pass-through? Why didn’t Citrix enable that feature for XenApp as well?
HDX3D Pro with desktop GPU Written by Mark on 2010-11-16 19:35:41Thomas:
Thanks for sharing your experience with HDX3D Pro with VM. It makes the Citrix guide a lot easy to follow.
When you run HDX3DConfigCmdLineX86.exe Display Current_options, do you see CODEC as CPU or GPU?
I know FX4800 is a supported Card but the price is rather high. Have you try desktop counterpart GTX280? The price is a lot cheaper using those card.
multi GPU pass-through is not a cheap solution. It will be very interesting to see how poeple using it.
Thanks again for the article!
HDX3D Pro with desktop GPU Written by Thomas Koetzing on 2010-11-16 19:29:07>I know FX4800 is a supported >Card but the price is rather high. And is really not a problem for those who use professional apps like CATIA
>Have you try desktop counterpart GTX280? >The price is a lot cheaper using those card. You can not try any card. The requirements are very specific and you need CUDA and SLI Multi-OS. So using just a card with multi GPU is not enough.
>multi GPU pass-through is not a cheap solution For those who want to run single blade PC with Multi-GPU cards it's not cheap anyway. Beeing able to virtulaize is a big benfit for them!
Written by Guest on 2010-11-17 15:56:43Just a small remark: there are other hypervisors that support multicore in a vm - vSphere 4.1 supports it (but no GPU passthrough to my knowledge)
GPU with vShere Written by Thomas Koetzing on 2010-11-17 16:05:43>vSphere 4.1 supports it I know but it's also a more or less hidden feature and that's why I wrote not officially supported.
Aero in XenDesktop 4 Written by Gast on 2010-11-18 13:30:57Is it correct, that I can´t use Aero in a Win7 Machine running on XenDesktop 4 with XenServer 5.6 as hypervisor?
Having some trouble in my XenDesktop Test-Environment. Some of my users have problems with slow mouses when using the Windows Standard Theme. When they switch there theme to Windows Classic, the problems are resolved. Any Ideas where this comes from?
Thanks up foreward!
Aero in XenDesktop 4 Written by Thomas Koetzing on 2010-11-18 13:41:11>can´t use Aero in a Win7 Machine running on >XenDesktop 4 with XenServer 5.6 as hypervisor? The correct statement is that you can NOT use Aero with ANY Hypervisor unless you have direkt access to the Graphic Card. Aero does work as I described with HDX 3D Pro Graphics and GPU Pass-through in XenDesktop.
Areo is not supported by Microsoft on any Hypervisor and you would need GPU Pass-through with XenServer or Hyper-V Remote FX (available with SP1 for Server 2008 R2)
Written by Gast on 2010-11-18 14:27:31Many Thanks for your quick and as always very clear answer!
Do you have a clue about the second part of my question? I have users with HP EliteBooks running Win7 using our XD running Win7 and users with old desktop hardware running WinXP using our XD running Win7, that have the problem with slow and de:ruckeligen mouses. On the other hand, I have users with the same hardware, that do not have those problems. Any suggestions (perhaps aero emulation by the CPU of the XD causing this)?
client access Written by Thomas Poppelgaard on 2011-02-18 15:48:33We have a Dell R5400 server with 2xFX Quadro 4000.
XenServer 5.6 Fp1 is installed, and i have enable GPU Sharing enabled and verified. with zgrep -e '^CONFIG_XEN_PCIDEV_BACK' /proc/config.gz
also with lspci i can verify the two graphic cards.
I have created one virtual machine (windows 7x64,6gbram) I have assigned 1 fysical graphic card to the virtual machine
When i boot up the virtual machine, i can see 2 VGA cards. (1 card is the virtual vga, which all virtual machines gets, and then i can see the fysical graphic card as a VGA, because there is no driver loaded)
I then installed the official Nvidia driver FX Quadro 4000 driver, rebooted the machine. Then the virtual machine comes up with a blank screen (white) and dosent start further up, i can verify that because under network settings the virtual machine dosent get any Ip adress.
Do you have any surgestions ?
Hi Written by Guest on 2011-04-16 06:57:54Did you find any solution yet Thomas P.? We are running into the same problem.
VM problem Written by Guest on 2011-04-16 06:57:18Hello Thomas P. Did you find any resolution for your problem. I am facing the exact same problem.
Hope to hear from you
Dell R5400 Written by Thomas Poppelgaard on 2011-04-20 10:48:13Hello,
the DELL R5400 can't support GPU Passthrouth, there are not all VT-D functions available
Can I use more than one GPU per user? Written by Guest on 2011-04-22 19:47:07Hello Thomas,
Thank you for the great info. With SLI enabled, does that mean that I can configure two cards in an SLI config and have Citrix see it as one card?
Thank you! Chris
Can't staring W7 VM,after install Drive Written by Guest on 2011-05-04 17:19:54Hi,Thomas I want try "xenServer 5.6 multi GPU pass-through for XenDesktop HDX 3D Pro Graphics" Function. cant' Starting W7 VM,after Install FX3800 (Nvidia) Driver . Boot the vm, staring windows ...then blue screen.
Can't staring W7 VM,after install Drive Written by Thomas Koetzing on 2011-05-04 17:26:11Note: XenDesktop 5.0 does NOT support HDX 3D Pro Graphics with the current release (next will). BSOD is driver related and what STOP error did you get?
Can't staring W7 VM,after install Drive Written by Guest on 2011-05-04 18:22:05I get this attempt to reset the display driver and recover from timeout failed.
Technical information: Stop: 0x00000116
Can't staring W7 VM,after install Drive Written by Guest on 2011-05-04 18:35:41Hi,Thomas Is a nvlddmkm.sys Error
Can't staring W7 VM,after install Drive Written by Guest on 2011-05-08 18:21:22nvlddmkm.sys error in win7-64bit and winxp-32bit Try many nvidia driver..all is error..
Keine nVidia-GPU in der Anzeige Written by Gast on 2011-11-07 12:23:08Hi Thomas, hast Du mir einen Hinweis darauf, warum ich die Grafikkarte zwar installieren, aber nicht in den Anzeige-Einstellungen unter Windows 7 auswählen kann? Bekomme nur den Citric Monitor angezeigt, und somit funktioniert VectorWorks natürlich recht schwerfällig.