About Virtual GPU

[xiaoweishang]

NVIDIA Open GPU Kernel Modules Version

515.48.07

Operating System and Version

Linux

Kernel Release

5.x

Build Command

Hi,

I wonder whether you have open sourced the feature of "NVIDIA virtual GPU (VGPU)"? [1] says it will be open sourced later but I find some code related to VGPU in open-gpu-kernel-modules (version 515.48.07), such as "open-gpu-kernel-modules/src/nvidia/kernel/vgpu/". Is the current VGPU related code in 515.48.07 only for the functionality of GPU pass-through while using GPUs in virtual machines? Thanks!

[1] https://us.download.nvidia.com/XFree86/Linux-x86_64/515.48.07/README/kernel_open.html

Terminal output/Build Log

N/A

More Info

N/A

[mtijanic]

Hi there. The currently published code does not support VGPU features. You did rightly note that we have many references to VGPU in the codebase, but this is usually because the infrastructure for Guest<->Host communication in VGPU was used as the basis for CPU<->GSP communication as well.

In other cases, because the source comes from a shared codebase, it might have just been too problematic to remove dead IS_VIRTUAL(pGpu) cases from the code.

We do hope to eventually get full feature parity between the open and proprietary drivers, including VGPU support, but I don't have any timelines to share here yet.

Thanks.

[NormanGu]

Not sure, but I don't think it uses mdev exactly. However it's not really relevant, there's an escape from guest kernel to host userspace somehow, and then host userpsace makes a call to host kernel to execute a procedure and propagate the result all the way back.

It's arguable whether or not that satisfies the remote-ness requirements to be worthy of the "RPC" name, but it is the name that we've been using for it since the start. And, the GSP interface was derived from that.

thanks a lot！
I find a VGPU send rpc related function：_rpcSendMessage_VGPUGSP，i dig into this function and looks like it will call into RpcVgpuGspRingDoorbell function in the end, which is not inplement in TU10X and GA100, so i guess VGPU also use rpc to interact with gsp.

[mtijanic]

There are three different possible implementations of rpcSendMessage(), but only two are open source. The _VGPUGSP variant is used on Ada+ and requires a VGPU component to be running on the GSP as well. On regular non-virtualized GSP-enabled systems, we use _kgspRpcSendMessage() for CPU->GSP communication.

Then, the closed source driver also has a third _rpcSendMessage_VGPU() implementation that does the Guest->Host escape entirely on the CPU. This model of VGPU is not supported on the open drivers.

[NormanGu]

There are three different possible implementations of rpcSendMessage(), but only two are open source. The _VGPUGSP variant is used on Ada+ and requires a VGPU component to be running on the GSP as well. On regular non-virtualized GSP-enabled systems, we use _kgspRpcSendMessage() for CPU->GSP communication.

Then, the closed source driver also has a third _rpcSendMessage_VGPU() implementation that does the Guest->Host escape entirely on the CPU. This model of VGPU is not supported on the open drivers.

thanks, it helps me a lot!
so I guess _rpcSendMessage_VGPUGSP is supposed to be execute on Host, and _rpcSendMessage_VGPU which is not open source is supposed to be execute on Guest, and _rpcSendMessage_VGPU will final escape to _rpcSendMessage_VGPUGSP, is this correct?

[mtijanic]

Sorry I wasn't clear before, it's:

• _kgspRpcSendMessage() Host -> GSP
• _rpcSendMessage_VGPUGSP() Guest -> GSP
• _rpcSendMessage_VGPU() Guest -> Host (legacy path, closed source only)

We don't support the legacy path on the GSP driver model. Theoretically it would be possible to do Guest->Host, and then forward Host->GSP to support it, but AFAIK we never productized that.

[NormanGu]

Sorry I wasn't clear before, it's:

• _kgspRpcSendMessage() Host -> GSP
• _rpcSendMessage_VGPUGSP() Guest -> GSP
• _rpcSendMessage_VGPU() Guest -> Host (legacy path, closed source only)

We don't support the legacy path on the GSP driver model. Theoretically it would be possible to do Guest->Host, and then forward Host->GSP to support it, but AFAIK we never productized that.

OK，I check the code again, the call stack like _rpcSendMessage_VGPUGSP() --> RpcVgpuGspRingDoorbell -->GPU_VREG_WR32, the GPU_VREG_WR32 function calculate address with virtualRegPhysOffset +address，and virtualRegPhysOffset is get from gpuGetVirtRegPhysOffset_TU102(with macro NV_VIRTUAL_FUNCTION_FULL_PHYS_OFFSET), so, when different GUEST calls into RpcVgpuGspRingDoorbell, they are actually writing into the same offset (addr+virtualRegPhysOffset), I mean, at least 'SAME' offset in their own address view, but how can gsp seperate different GUESTs from a same reg address access? unless the same offset in GUEST view trafic into different physic address in physic GPU, that means host may trap this offset, or using SRIOV