vGPU in an LXC container¶

no longer used

i no longer run vGPU on the cluster.

How to get working in a privileged container¶

wow this one is hard.... you can avoid the id mapping stuff by not using a privileged container...

Assumptions:¶

you have a debian 12 container, you added the non-free deb and have installed the non-free drivers as per the host instructions
you have run cat /etc/groups in the container and noted down the GID for render (lets call that CTRGID) and gid for video (lets call that CTVGID).
you have run cat /etc/groups in the container and noted down the GID for render (lets call that HSTRGID) and gid for video (lets call that HSTVGID).
that you have va info fully working

Create Container¶

create container privileged, with debian 12, starts it
apt update, apt upgrade, install non free drivers, vainfo and intel_gpu_top tools
add root to user and video groups (this will mean when we get to ID mapping you don't need to tart about with user mappings - only group ones)
```
usermod -a -G render root
usermod -a -G video root
```
shutdown container

Edit container conf file¶

These are stored in /etc/pve/lxc and have the VMID.conf anme
nano /etc/pve/lxc/VMID.conf

Add lxc device mapping¶

Here you add a line for the card uyou want and the rendere. Note if you map a VF (card) to a container it means that is hard mapped, if you have that VF in a pool for VMs please remove it from the pool (this means also these containers cannot be HA)

In the example below i chose card6 - which is renderD134 These are mapped into the container as card0 and renderD128 Change your numbers as per your own VF / card mappings

lxc.cgroup2.devices.allow: c 226:6 rwm
lxc.mount.entry: /dev/dri/card6 dev/dri/card0 none bind,optional,create=file

lxc.cgroup2.devices.allow: c 226:134 rwm
lxc.mount.entry: /dev/dri/renderD134 dev/dri/renderD128 none bind,optional,create=file

Add ID mapping (only needed in unprivileged)¶

add the following... and here it gets complex as it will vary based on the numbers you recorded earlier - let me try... the aim is to have a continguois block of mappings but the syntax is um difficult...

lxc.idmap: u 0 100000 65536
lxc.idmap: g 0 100000 CTVGID
lxc.idmap: g CTVGID HSTVGID 1
lxc.idmap: g CTVGID+1 1000{CTVGID+1} CTRGID-CTVGID-1
lxc.idmap: g CTRGID HSTVGID 1
lxc.idmap: g CTRGID+1 100{CTRGID+1} 65536-{CTRGID+1}

so as an example, these are my values:

        host > ct
video:    44 > 44
render:  104 > 106

this is what i added to my VMID.conf file (in my case /etc/pve/lxc/107.conf

lxc.idmap: u 0 100000 65536
lxc.idmap: g 0 100000 44
lxc.idmap: g 44 44 1
lxc.idmap: g 45 100045 61
lxc.idmap: g 106 104 1
lxc.idmap: g 107 100107 65429

add your two CT values to nano /etc/subgid (only needed in unprivileged)

in my case:

root:106:1
root:44:1

after this you should be able to start up the container and run vainfo and perform transcoding.

check permissions with ls -la /dev/dri it should look like this:

root@vGPU-debian-test:~# ls -la /dev/dri
total 0
drwxr-xr-x 2 root   root         80 Oct  7 00:22 .
drwxr-xr-x 7 root   root        500 Oct  7 00:22 ..
crw-rw-rw- 1 nobody video  226,   0 Oct  4 21:42 card0
crw-rw-rw- 1 nobody render 226, 128 Oct  4 21:42 renderD128

if the group names do not say video and render then you did something wrong

Note: YYMV

For example plex HW transcoded just fine on my system.

Emby on the otherhand seems to interrogate the kernel driver directly and gets the wrong answers - this is IMHO an issue with their detection logic not supporting this scenario.

Another example is intel_gpu_top which doesn't seem to work in this mode either - this is because it only works with the PMUs not the VFs (so somoene said)

Or maybe i just have no clue what i am doing, lol.

---work in progress 2023.10.6---

This page started as a gist: the original, with its comments