GPU on Charlie Chiang's blog

Proxmox VE Shared GPU Installation Guide

Sat, 16 Aug 2025 20:49:00 +0800

Introduction

Most schools and universities in China have a limited number of GPU servers (often only one or two, or even none). This makes the management of GPU development machines quite different than corporation which have virtually unlimited GPU servers.

Let’s consider a normal lab environment with 12 students, each needing a GPU server for their projects, and we have one GPU server with 8 GPUs available.

If we give each student a dedicated GPU server, we would need 12 servers, this is impractical and costly.
If we let all the students share a single GPU server (all students have access to all resources), it would be very difficult to manage, as each student would need to install their own software and dependencies, leading to conflicts. Soon the GPU server will become a hot mess. Making each student use a different user account would not solve the problem, as they would still share the same OS and software environment, and will potentially break each other’s environment.
If we make use of IOMMU and PCIe passthrough, we can assign a virtual machine with one dedicated passthrough’d GPU to each student. This approach has almost no interference between students, as each student has their own OS and software environment, even with it’s own GPU. However, this leads to a lot of wasted resources, because most of the time the GPU is idle, and each student will only have access to one GPU at a time, even if all of the GPUs are idle.
If we use GPU virtualization, we can assign a virtual machine with a shared GPU to each student. This way, each student can use the GPU resources as needed, and the GPU can be shared among multiple students. This approach is more efficient and cost-effective, as it allows for better resource utilization. However, it requires a GPU that supports virtualization, such as NVIDIA’s vGPU or AMD’s MxGPU. Most consumer GPUs do not support virtualization, so this approach is not feasible for most schools and universities.

This is where containerization comes in. By using containerization, we can create a shared GPU development server that runs a container for each student. Each container can have its own software environment, and all (or some of) of the GPUs can be shared among multiple containers. This approach is more efficient and cost-effective, as it allows for better resource utilization, and does not require a GPU that supports virtualization. However, it should be noted that this approach is not as isolated as virtual machines, as all containers share the same kernel and GPU resources. Therefore, it is important to ensure that the containers are properly configured, and that the students are aware of the limitations and potential issues that may arise from sharing the same GPU resources (for example, if one student runs a GPU-intensive task, it may affect the performance of other students’ containers).

We will be using the CT (LXC Containers) in Proxmox VE to achieve this. Why not use Docker? Because Docker is meant for running applications, not for running full Linux distros. Although you can use something like sysbox to run a full Linux distro in Docker, i would still prefer to use LXC Containers, as they are built for this purpose and are already built into Proxmox VE, making it easier to manage and deploy.

Proxmox VE Installation

Download the latest Proxmox VE ISO from the official website. At the time of writing, the latest version is Proxmox VE 9.0. Use whatever method you prefer to install Proxmox VE, such as using a USB drive.

I will use IPMI of the server to install Proxmox VE, as it is the most convenient method for me. You can also use a monitor and keyboard to install Proxmox VE if you prefer.

Mount the Proxmox VE installation ISO as virtual media so we can boot into.

Now reboot the server into the virtual media. Supermicro motherboards lets you invode the boot menu using F11. Your motherboard may have a different shortcut.

Choose our CDROM virtual media.

Boot into the Proxmox install menu (I prefer terminal UI over GUI ones).

After you accepted the license, you will need to choose the target installtion disk. You should choose the boot SSDs on your server, not data drives. The drive in picture is 3*1.92TB SSDs in RAID 5.

Toggle Advanced options. Keep ext4, we will not be using ZFS. I know ZFS has CoW, snapshotting, compressing, checksums, and a ton of other features. But we will not use ZFS, because it will potentially cause some problems, e.g., high IO load when using RAID-Z zvols, extremely slow container start times when using ZFS storage driver in Docker, SSD write amplifications, low SSD random read/write performance, ZFS ARC not being given back to the OS as fast as needed on high memory pressure systems leading to OOM, and a ton of other issues that I previously encountered. To save me some trouble, I will use the battle-tested ext4.

Total size: keep as-is
Swap size: 0, we don’t need it, our memory is large enought (1TiB) and we will later use ZRAM as swap
Maximum root volume size: 100 (GiB), make it slightly larger so we can install things into the root volume, but not too large to occupy the data volume space.

Other options can be left empty.

Other install steps (Keyboard setup, root password) can proceed as you normally would do. Remember to set a complex root password.

Regarding to IP addresses:

If you are going to use a static IP address like I do, you can just set it here and forget about it.

If you are going to use a dynamic (DHCP) addresses, you should keep what’s already in here (static IP). You don’t need to change anything. The valid IP address should already be automatically discovered from DHCP servers and filled in. After the installation, you can change the static IP mode to DHCP mode by running vi /etc/network/interfaces and make the following edits:

1
2
3
4
5
6
7
8


 auto vmbr0
- iface vmbr0 inet static
+ iface vmbr0 inet dhcp # Enable DHCP and remove static IP.
- address 10.112.154.220/16
- gateway 10.112.0.1
 bridge-ports eno1
 bridge-stp off
 bridge-fd 0

After you enabled DHCP in the config, apply it by ifreload -a. Check if it takes effect by ip a show vmbr0, you should see something like valid_lft 7160sec preferred_lft 7160sec. If there is no xx sec inside it, your DHCP is malfunctioning.

1
2
3
4
5
6
7


root@z8:/etc/network# ip a show vmbr0
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
 link/ether 30:13:8b:6d:3a:a6 brd ff:ff:ff:ff:ff:ff
 inet 10.112.154.220/16 brd 10.112.255.255 scope global dynamic vmbr0
 valid_lft 7160sec preferred_lft 7160sec
 inet6 fe80::3213:8bff:fe6d:3aa6/64 scope link
 valid_lft forever preferred_lft forever

Proxmox VE Post-Installtion

Change APT Sources

In China, we don’t really have a good global Internet connection, so we will change APT source to mirrors in China.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# Change APT source mirrors
sed -i 's/deb.debian.org/mirrors.ustc.edu.cn/g' /etc/apt/sources.list.d/debian.sources
sed -i 's/security.debian.org/mirrors.ustc.edu.cn/g' /etc/apt/sources.list.d/debian.sources

# Change Ceph source mirrors
if [ -f /etc/apt/sources.list.d/ceph.sources ]; then
 CEPH_CODENAME=`ceph -v | grep ceph | awk '{print $(NF-1)}'`
 source /etc/os-release
 cat > /etc/apt/sources.list.d/ceph.sources <<EOF
Types: deb
URIs: https://mirrors.ustc.edu.cn/proxmox/debian/ceph-$CEPH_CODENAME
Suites: $VERSION_CODENAME
Components: no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF
fi

Remove enterprise repo and add no-subscription repo.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Remove enterprise sources
rm /etc/apt/sources.list.d/pve-enterprise.sources

# Add no-subscription sources
cat > /etc/apt/sources.list.d/pve-no-subscription.sources <<EOF
Types: deb
URIs: https://mirrors.ustc.edu.cn/proxmox/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF

Change CT Template Sources

Use mirrors in China.

1
2


sed -i.bak 's|http://download.proxmox.com|https://mirrors.ustc.edu.cn/proxmox|g' /usr/share/perl5/PVE/APLInfo.pm
systemctl restart pvedaemon

Stop Cluster Services

We are not using PVE clusters. Disable them. If you are using PVE clusters, you probably will not be reading this guide :P

1
2
3


systemctl disable --now pve-ha-crm.service
systemctl disable --now pve-ha-lrm.service
systemctl disable --now corosync.service

Install Common Tools

Useful tools that will be used regularly.

1

apt install htop sysstat vim sudo

Configure Shell

I like ZSH and my dotfiles, so I will use them.

1

apt install git zsh

1
2
3
4
5
6
7
8


cd
git clone --depth=1 https://github.com/charlie0129/dotfiles.git
cd dotfiles
./bootstrap.sh -f
chsh -s /usr/bin/zsh

# Run zsh and follow the instructions
zsh

Configure ZRAM Swap

1
2
3
4
5


git clone --depth=1 https://github.com/foundObjects/zram-swap.git
cd zram-swap
./install.sh
cd ..
rm -rf zram-swap

The default lz4 is a good balance between speed and compression ratio. If you want to sacrifice speed for better compression, you can change the config to use zstd as the compression method.

1

sed -i 's/_zram_algorithm=.*/_zram_algorithm="zstd"/g' /etc/default/zram-swap

Apply

1

systemctl restart zram-swap

Tune kernel parameters to make better use of ZRAM

1
2
3
4
5
6


cat <<EOF > /etc/sysctl.d/zram.conf
vm.swappiness = 180
vm.watermark_boost_factor = 0
vm.watermark_scale_factor = 125
vm.page-cluster = 0
EOF

Apply

1

sysctl --system

Enable IPv6 SLAAC

Proxmox disables IPv6 by default. To enable IPv6 SLAAC:

1
2
3
4
5
6
7
8


cat <<EOF > /etc/sysctl.d/ipv6.conf
net.ipv6.conf.default.accept_ra = 2
net.ipv6.conf.all.accept_ra = 2
net.ipv6.conf.default.forwarding = 1
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.proxy_ndp = 1
net.ipv6.conf.all.proxy_ndp = 1
EOF

Apply it

1
2
3


sysctl --system
# Note: the following command may break your internet connection.
systemctl restart networking

TRIM Optimizations

TRIM will help prolong SSD lifespan and maintain performance. Enable it on the LVM thin pool.

1

sed -i 's/.*issue_discards = .*/\tissue_discards = 1/g' /etc/lvm/lvm.conf

Since LXCs are just on the host, it will automatically issue discards if the underlying storage supports it, so no further action is needed. If you are using VM, be sure to use SCSI disks (VirtIO SCSI Single) with discard=on option. I’ve heard that VirtIO Block devices will not work with dicards but I haven’t tested it yet. Correct me if I’m wrong.

Limit Journal Size

1

sed -i 's/.*SystemMaxUse.*/SystemMaxUse=32M/g' /etc/systemd/journald.conf

Apply

1
2


systemctl daemon-reload
systemctl restart systemd-journald

NVIDIA P2P Driver Installation

Instead of regular NVIDIA drivers, we will be installing P2P-enabled drivers to force enable PCIe P2P capabilities on consumer cards (like GeForce RTX 4090). This will bring performance boost (~10%) across multiple scenarios. For details, refer to my blog post: Enabling PCIe P2P on NVIDIA RTX 4090s

Disable IOMMU

Since PCIe P2P in Linux doesn’t work so well with IOMMU-enabled systems (there are many potential issues you may run into), you may as well just disable IOMMU. Note that if you have more than 255 CPU cores, you will only have access to 255 CPUs due to APIC fallback.

To do disable IOMMU, you can either

Disable Intel VT-d (AMD is on by default) in BIOS
Disable intel_iommu or amd_iommu in Linux kernel parameters

To disable intel_iommu or amd_iommu in Linux kernel parameters, use the following command to add intel_iommu=off and amd_iommu=off to your GRUB_CMDLINE_LINUX_DEFAULT. BTW: I also reduce the screen resolution to 1024x768 because it’s a server

1

sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=off amd_iommu=off video=1024x768@60"/g' /etc/default/grub

Reboot your system to see the effect.

1

reboot

Make sure the follow command produces NO entries. If there are output, it means IOMMU is not correctly disabled.

1

ls /sys/class/iommu

Disable ACS

You may need to disable ACS to get PCIe P2P to work, refer to https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html (PCI Access Control Services)

Enable Above 4G Decoding

PCIe P2P will need to access each GPU’s memory, so the PCIe BAR should be large enough to cover GPU’s memory. You must enable Above 4G Decoding in your motherboard settings to make the PCIe BAR large enough to work. Enable it in your motherboard’s BIOS.

Install Official Driver

Before we install the P2P kernel module, we need to install the official drivers.

Since we are installing unofficial P2P kernel modules, there is a limited number of driver version that will work. The version of the P2P kernel modules MUST be the same as the official driver we are going to install.

Refer to https://github.com/tinygrad/open-gpu-kernel-modules to see what versions are available. I also have my patches that works with much newer driver versions (575.57.08) (at the time of writing) https://github.com/charlie0129/open-gpu-kernel-modules .

I will be using version 575.57.08, so we should download the official driver of the same version (575.57.08). You should download the .run file, e.g., NVIDIA-Linux-x86_64-575.57.08.run

1
2
3


# Assume you download the installer as NVIDIA-Linux-x86_64-575.57.08.run
installer="NVIDIA-Linux-x86_64-575.57.08.run"
chmod +x $installer

Skip kernel modules because we will install P2P-patched version later.

1

./$installer --no-kernel-modules

Choose whatever you want, doesn’t matter.

Yes, disable nouveau, we will be using NVIDIA drivers.

Do not abort, just continue. Nouveau will be disabled on next boot.

Just choose the default option for every step that comes later.

Remove the driver installation file because we will not be using it later.

1

rm $installer

You can skip reboot now. We will reboot after we installed the kernel modules.

Install P2P-Enabled Kernel Modules

Make sure the version of the P2P-enabled kernel modules match the version of the driver. I will use veriosn 575.57.08.

1

git clone --depth=1 -b 575.57.08-p2p https://github.com/charlie0129/open-gpu-kernel-modules.git

To build kernel modules, install build dependencies and kernel source. sudo installed because the install script uses sudo but we don’t have it now.

1

apt install sudo build-essential proxmox-headers-$(uname -r)

Build and install the kernel module

1

./install.sh

You can safely ignore the NVIDIA-SMI failure as long as the build succeeds (you should see a DEPMOD /lib/modules/6.14.8-2-pve line) because we haven’t rebooted yet (so NVIDIA-SMI can’t be used).

Remove the source after installing

1
2


cd ..
rm -rf open-gpu-kernel-modules

Reboot your system and you should see nvidia-smi running fine.

To tell if P2P is enabled, we will use a simple method (performance testing will be done later):

1

nvidia-smi -q | grep -i bar -A 3

You should see > 2048 MiB BAR1 Total memory, 32768 MiB in my case.

1
2
3
4


 BAR1 Memory Usage
 Total : 32768 MiB
 Used : 2 MiB
 Free : 32766 MiB

Enable Persistence Mode

Persistence mode will:

Keep the GPU driver running so program can start faster
Lower GPU power mode when it’s idle to save power. For RTX 4090s, it can drop from ~70W to ~10W.
Make sure /dev/nvidia* device nodes are ready. This is useful because we are passing the GPU devices /dev/nvidia* to CTs (LXC Containers) later, it requires the device nodes to be present on start up for auto-started CTs.

So you really should enable it.

To enable persistence mode:

1

/usr/bin/nvidia-smi -pm 1

To make it persistent across reboots, we will use a cron job.

1

crontab -e

Add a line to run nvidia-smi on boot:

1

@reboot /usr/bin/nvidia-smi -pm 1 >/dev/null 2>&1

Why not use nvidia-persistenced systemd service? Because for whatever reason, it does not initialize the device nodes /dev/nvidia* correctly, so auto-started CTs will encounter Cuda failure 'unknown error'.
1
2
3
4
cd /usr/share/doc/NVIDIA_GLX-1.0/samples
tar jxf nvidia-persistenced-init.tar.bz2
cd nvidia-persistenced-init
./install.sh

You should feel nvidia-smi runs much faster than before.

Build CT Templates

We will build a CT Template that has everything a student will need:

Common build dependencies
CUDA
Docker
…

Download the Linux distro you want. I chose Ubuntu 24.04, not because I like Ubuntu (I use Debian), but because most students only know Ubuntu.

Create an unprivileged CT just like you normally would. Just remember to give it a bit more disk space (32GB or more) because we will install CUDA later and CUDA is really large.

Install Docker

After you created the CT, shut it down. Add additional settings to allow Docker to use overlayfs driver, otherwise Docker images will be extremely inefficient in CTs.

1
2
3
4
5
6
7


# This command should be run on the host, not CT.
# Assume 8000 is the ID of your template CT
cat <<EOF >> /etc/pve/local/lxc/8000.conf
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
EOF

All following commands should be run in the CT unless specified otherwise.

Add Docker configuration. Systemd journal is used as log driver to make use of previously configured journal size limit. By writing logs to a centralized location, it’s also useful if you want to write logs to memory to reduce disk writes (e.g. log2ram).

1
2
3
4
5
6
7
8


mkdir -p /etc/docker/
cat <<EOF > /etc/docker/daemon.json
{
 "live-restore": true,
 "experimental": true,
 "log-driver": "journald"
}
EOF

Install Docker

1
2


export DOWNLOAD_URL=https://mirrors.ustc.edu.cn/docker-ce
curl -fsSL https://get.docker.io | sh

Revert Containerd Config

Docker modified containerd config. We will revert it to the default config.

1

containerd config default >/etc/containerd/config.toml

Add GPUs to CT

Choose which GPU you need to passthrough by looking for the index in nvidia-smi, or by ls -l /dev/nvidia*

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# This command should be run on the host, not CT.
# ls -l /dev/nvidia*
crw-rw-rw- 1 root root 195, 0 2025-08-17 01:47:58 /dev/nvidia0
crw-rw-rw- 1 root root 195, 1 2025-08-17 01:48:00 /dev/nvidia1
crw-rw-rw- 1 root root 195, 2 2025-08-17 01:48:01 /dev/nvidia2
crw-rw-rw- 1 root root 195, 3 2025-08-17 01:48:02 /dev/nvidia3
crw-rw-rw- 1 root root 195, 4 2025-08-17 01:48:03 /dev/nvidia4
crw-rw-rw- 1 root root 195, 5 2025-08-17 01:48:05 /dev/nvidia5
crw-rw-rw- 1 root root 195, 6 2025-08-17 01:48:06 /dev/nvidia6
crw-rw-rw- 1 root root 195, 7 2025-08-17 01:48:07 /dev/nvidia7
... (omitted)

For example, if want to add all 8 GPUs (0, 1, 2, 3, 4, 5, 6, 7) to CT, I will need to run:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# This command should be run on the host, not CT.
# Assume 8000 is the ID of your template CT
cat <<EOF >> /etc/pve/local/lxc/8000.conf
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia1 dev/nvidia1 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia2 dev/nvidia2 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia3 dev/nvidia3 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia4 dev/nvidia4 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia5 dev/nvidia5 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia6 dev/nvidia6 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia7 dev/nvidia7 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/fb0 dev/fb0 none bind,optional,create=file
EOF

Note the lxc.mount.entry: /dev/nvidiaX dev/nvidiaX none bind,optional,create=file lines. Each line represents a single GPU to add to the CT. I will add 8 GPUs, so I add 8 lines from nvidia0 through nvidia7. Remove or add them as needed.

Check if the GPUs are there inside CT:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# ls -l /dev/nvidia*
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:00 /dev/nvidia-modeset
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:50:43 /dev/nvidia-uvm
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:50:43 /dev/nvidia-uvm-tools
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:47:58 /dev/nvidia0
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:00 /dev/nvidia1
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:01 /dev/nvidia2
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:02 /dev/nvidia3
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:03 /dev/nvidia4
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:05 /dev/nvidia5
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:06 /dev/nvidia6
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:07 /dev/nvidia7
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:47:58 /dev/nvidiactl

We haven’t installed drivers yet, so nvidia-smi is not available.

Install GPU Driver

Important: you should use the same driver version as the host (575.57.08 in my case).

1
2
3
4


wget <driver-runfile-url> -O driver.run
chmod +x driver.run
driver.run --no-kernel-modules
rm driver.run

Just install as normal.

Now you should be able to use nvidia-smi and see the GPUs you added to the CT (8 GPUs in my case).

Install CUDA

Important: current driver version MUST be greater than (or equal to) the driver required by CUDA. For example, CUDA 12.9.0 will work on my machine because CUDA 12.9.0 requires 575.00, and my current driver version is 575.57.08 (greater than 575.00).

1
2
3
4


wget <cuda-runfile-url> -O cuda.run
chmod +x cuda.run
cuda.run
rm cuda.run

Remember to deselect the driver (we already installed the correct driver):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


┌──────────────────────────────────────────────────────────────────────────────┐
│ CUDA Installer │
│ - [ ] Driver <== deselect this │
│ [ ] 575.51.03 <== deselect this │
│ + [X] CUDA Toolkit 12.9 │
│ [ ] CUDA Demo Suite 12.9 <== deselect this (not useful, takes up space) │
│ [ ] CUDA Documentation 12.9 <== deselect this (not useful, takes up space) │
│ - [ ] Kernel Objects │
│ [ ] nvidia-fs │
│ Options │
│ Install │
│ │
│ Up/Down: Move | Left/Right: Expand | 'Enter': Select | 'A': Advanced options │
└──────────────────────────────────────────────────────────────────────────────┘

After CUDA installation, configure PATH and LD_LIBRARY_PATH per printed instructions.

1
2
3
4
5


... (omitted)
Please make sure that
 - PATH includes /usr/local/cuda-12.9/bin
 - LD_LIBRARY_PATH includes /usr/local/cuda-12.9/lib64, or, add /usr/local/cuda-12.9/lib64 to /etc/ld.so.conf and run ldconfig as root
... (omitted)

Note that you can use /usr/local/cuda instead of /usr/local/cuda-XX.X so it’s independent of CUDA versions.

Also add export CUDA_HOME=/usr/local/cuda to your shell rc file.

nvcc should be available after a shell reload:

1
2
3
4
5
6


# nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Apr__9_19:24:57_PDT_2025
Cuda compilation tools, release 12.9, V12.9.41
Build cuda_12.9.r12.9/compiler.35813241_0

Install NCCL

You will need NCCL for most GPU-related communications.

You should find a NCCL that’s compatible with your CUDA version. For example, NCCL 2.27.3 is compatible with my CUDA version 12.9.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


# Download NCCL package
wget https://xxx/nccl_2.27.3-1+cuda12.9_x86_64.txz

tar Jxf nccl_2.27.3-1+cuda12.9_x86_64.txz

# Set some variables
NCCL_VERSION=2.27

mkdir -p /usr/local/nccl-$NCCL_VERSION

cp -vRf nccl_2.27.3-1+cuda12.9_x86_64/* /usr/local/nccl-$NCCL_VERSION
rm -rf nccl_2.27.3-1+cuda12.9_x86_64

ln -s /usr/local/nccl-$NCCL_VERSION /usr/local/nccl

echo "/usr/local/nccl/lib" >>/etc/ld.so.conf
ldconfig

ln -s /usr/local/nccl/include/nccl.h /usr/local/include/nccl.h

# Remove NCCL package
rm https://xxx/nccl_2.27.3-1+cuda12.9_x86_64.txz

Also add export NCCL_HOME=/usr/local/nccl in your shell rc file,

Install NVIDIA Container Toolkit

To run Docker containers with GPUs.

Note that the URLs below are mirrors in China. You can use original URL if you prefer.

1
2
3
4
5
6
7
8


curl -fsSL https://mirrors.ustc.edu.cn/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://mirrors.ustc.edu.cn/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
 sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
 tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sed -i 's/nvidia.github.io/mirrors.ustc.edu.cn/g' /etc/apt/sources.list.d/nvidia-container-toolkit.list

apt-get update
apt-get install -y nvidia-container-toolkit

Configure Docker and Containerd to use NVIDIA Container Toolkit

1
2


nvidia-ctk runtime configure --runtime=docker
nvidia-ctk runtime configure --runtime=containerd

Since we are running inside unprivileged containers, we need to set no-cgroups to true

1

nvidia-ctk config --set nvidia-container-cli.no-cgroups --in-place

Install NVIDIA Nsight Systems

So we can profile applications running on the GPU.

1
2
3


wget https://xxx/NsightSystems-linux-cli-public-2025.3.1.90-3582212.deb
dpkg -i NsightSystems-linux-cli-public-2025.3.1.90-3582212.deb
rm -f NsightSystems-linux-cli-public-2025.3.1.90-3582212.deb

Functionality Tests

NCCL

1
2
3
4


git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
git reset --hard 903918f # I only tested this commit.
make

P2P ON: NCCL_P2P_LEVEL=sys ./build/all_reduce_perf --minbytes 8 --maxbytes 128M --stepfactor 2 --ngpus 8 (change --ngpus accordingly)

P2P OFF: NCCL_P2P_DISABLE=1 ./build/all_reduce_perf --minbytes 8 --maxbytes 128M --stepfactor 2 --ngpus 8 (change --ngpus accordingly)

You should see a bandwidth increase after P2P is on.

CUDA P2P

1
2
3
4
5


git clone https://github.com/NVIDIA/cuda-samples.git
git reset --hard 9c688d7 # I only tested this commit.
cd cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest
make
./p2pBandwidthLatencyTest

You should see lower GPU-GPU latency, higher bandwidth if P2P is on.

Prepare for Templating

Check if there are unnecessary files and remove them.

1
2


cd
ls -l

Remove history

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


cd

unset HISTFILE
rm .*_history

echo -n >/var/log/lastlog
echo -n >/var/log/wtmp
echo -n >/var/log/btmp

journalctl --vacuum-time=1s

rm -f /var/log/*.log

Now, it’s ready for converting to a template. To give each student a new development container, just clone this template.

Other Minor Settings

RAID Card Write Back

To achieve maximum write speed, you can enable write back mode on your RAID card, so data is written to cache first. This will significantly (yes, by A LOT, sometimes over 100x) improve random write speed on HDDs (SSDs are already very fast). If your RAID card has a battery, you don’t need to worry about data integrity in case a power failure.

Write cache policy:

Write Throuh: no cache
Write Back: cached if battery backup is present on the RAID card
Always Write Back: always cached

Obstacles

Only 255 Cores are Recognized - x2APIC

I noticed that I have one CPU offline. It should recognize 256 CPUs but I only have 255 CPUs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# lscpu
Architecture: x86_64
 CPU op-mode(s): 32-bit, 64-bit
 Address sizes: 48 bits physical, 48 bits virtual
 Byte Order: Little Endian
CPU(s): 256
 On-line CPU(s) list: 0-254
 Off-line CPU(s) list: 255 <--------------- one CPU is offline -----------
Vendor ID: AuthenticAMD
 Model name: AMD EPYC 7763 64-Core Processor
 CPU family: 25
 Model: 1
 Thread(s) per core: 2
 Core(s) per socket: 64
 Socket(s): 2

After some digging, I learned that I need to enable x2APIC on both the motherboard and kernel to have more than 255 CPU cores recognized.

In BIOS Local APIC Mode should already be x2APIC .

Now I realized it can be because I disabled IOMMU for PCIe P2P before. It turns out that disableing IOMMU can limit the number of available logical cores to 255. The reason is that the Linux kernel disables x2APIC in this case and falls back to APIC, which can only enumerate a maximum of 255 (logical) cores.

I will keep IOMMU disabled because I need PCIe P2P to function (I don’t want to deal with P2P with IOMMU on). Just forget about the 256th core :p.

RTX4090 GPUDirect P2P Unlocked!

Fri, 07 Feb 2025 11:18:00 +0800

If you want see the test results directly, please jump to the Test the unlocked performance section.

What is P2P?

From NVIDIA: GPUDirect Peer to Peer enables GPU-to-GPU copies as well as loads and stores directly over the memory fabric (PCIe, NVLink). GPUDirect Peer to Peer is supported natively by the CUDA Driver.

As you can see, GPUDirect P2P allows for direct memory access between GPUs, bypassing the CPU and system memory. This can lead to significant performance improvements in multi-GPU systems. For example, in a multi-GPU LLM inference system, the data can be directly transferred between GPUs without involving the CPU, reducing latency and improving throughput.

Why is P2P disabled on RTX4090?

Easy, NVIDIA wants to make money. They want you to buy the more expensive Tesla GPUs if you want P2P.

Test machine configuration

CPU: 2 x AMD EPYC 7542 (32C 64T, 225W TDP, 2.9GHz base, 3.4GHz boost)
MEM: 2 x 8 x 32GB 3200MT/s DDR4 ECC
GPU: 8 x NVIDIA GeForce RTX 4090 (24GB GDDR6X, 450W TDP)

Topology (ideally, the GPUs should be behind a PCIe switch, but our test machine doesn’t have one):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


 GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS 24-31,88-95 3 N/A
GPU1 SYS X SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS 16-23,80-87 2 N/A
GPU2 SYS SYS X SYS SYS SYS SYS SYS SYS SYS SYS SYS 8-15,72-79 1 N/A
GPU3 SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS SYS 0-7,64-71 0 N/A
GPU4 SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS 56-63,120-127 7 N/A
GPU5 SYS SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS 48-55,112-119 6 N/A
GPU6 SYS SYS SYS SYS SYS SYS X SYS PHB PHB PHB PHB 40-47,104-111 5 N/A
GPU7 SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS 32-39,96-103 4 N/A
NIC0 SYS SYS SYS SYS SYS SYS PHB SYS X PIX PHB PHB
NIC1 SYS SYS SYS SYS SYS SYS PHB SYS PIX X PHB PHB
NIC2 SYS SYS SYS SYS SYS SYS PHB SYS PHB PHB X PIX
NIC3 SYS SYS SYS SYS SYS SYS PHB SYS PHB PHB PIX X

Legend:

 X = Self
 SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
 NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
 PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
 PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
 PIX = Connection traversing at most a single PCIe bridge
 NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

 NIC0: mlx5_0
 NIC1: mlx5_1
 NIC2: mlx5_2
 NIC3: mlx5_3

Test the original performance

PCI BAR

Only 256MB of BAR memory is available. Note that I did enabled Resizable BAR in the BIOS (not my fault).

1
2
3
4
5


# nvidia-smi -q | grep -i bar -A 3
 BAR1 Memory Usage
 Total : 256 MiB
 Used : 1 MiB
 Free : 255 MiB

NCCL Tests

Building NCCL Tests

1
2
3


git clone --depth=1 https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
make -j$(nproc)

Runnning:

Around 14.5GB/s of bandwidth.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


# ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 8
# nThread 1 nGpus 8 minBytes 8 maxBytes 134217728 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
# Rank 0 Group 0 Pid 1756513 on <REDACTED> device 0 [0x01] NVIDIA GeForce RTX 4090
# Rank 1 Group 0 Pid 1756513 on <REDACTED> device 1 [0x25] NVIDIA GeForce RTX 4090
# Rank 2 Group 0 Pid 1756513 on <REDACTED> device 2 [0x41] NVIDIA GeForce RTX 4090
# Rank 3 Group 0 Pid 1756513 on <REDACTED> device 3 [0x61] NVIDIA GeForce RTX 4090
# Rank 4 Group 0 Pid 1756513 on <REDACTED> device 4 [0x81] NVIDIA GeForce RTX 4090
# Rank 5 Group 0 Pid 1756513 on <REDACTED> device 5 [0xa1] NVIDIA GeForce RTX 4090
# Rank 6 Group 0 Pid 1756513 on <REDACTED> device 6 [0xc1] NVIDIA GeForce RTX 4090
# Rank 7 Group 0 Pid 1756513 on <REDACTED> device 7 [0xe1] NVIDIA GeForce RTX 4090
#
# out-of-place in-place 
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
 8 2 float sum -1 38.83 0.00 0.00 0 40.43 0.00 0.00 0
 16 4 float sum -1 38.73 0.00 0.00 0 38.59 0.00 0.00 0
 32 8 float sum -1 41.85 0.00 0.00 0 42.55 0.00 0.00 0
 64 16 float sum -1 42.55 0.00 0.00 0 42.65 0.00 0.00 0
 128 32 float sum -1 42.42 0.00 0.01 0 42.88 0.00 0.01 0
 256 64 float sum -1 42.44 0.01 0.01 0 42.53 0.01 0.01 0
 512 128 float sum -1 42.58 0.01 0.02 0 42.68 0.01 0.02 0
 1024 256 float sum -1 44.04 0.02 0.04 0 42.39 0.02 0.04 0
 2048 512 float sum -1 42.91 0.05 0.08 0 42.61 0.05 0.08 0
 4096 1024 float sum -1 43.18 0.09 0.17 0 42.69 0.10 0.17 0
 8192 2048 float sum -1 44.02 0.19 0.33 0 42.96 0.19 0.33 0
 16384 4096 float sum -1 43.10 0.38 0.67 0 42.99 0.38 0.67 0
 32768 8192 float sum -1 44.18 0.74 1.30 0 43.63 0.75 1.31 0
 65536 16384 float sum -1 45.66 1.44 2.51 0 45.28 1.45 2.53 0
 131072 32768 float sum -1 67.27 1.95 3.41 0 54.86 2.39 4.18 0
 262144 65536 float sum -1 124.8 2.10 3.68 0 121.1 2.17 3.79 0
 524288 131072 float sum -1 196.9 2.66 4.66 0 200.5 2.61 4.58 0
 1048576 262144 float sum -1 222.8 4.71 8.24 0 226.1 4.64 8.12 0
 2097152 524288 float sum -1 373.4 5.62 9.83 0 358.6 5.85 10.23 0
 4194304 1048576 float sum -1 580.8 7.22 12.64 0 592.0 7.09 12.40 0
 8388608 2097152 float sum -1 1041.8 8.05 14.09 0 1040.2 8.06 14.11 0
 16777216 4194304 float sum -1 1972.8 8.50 14.88 0 1979.5 8.48 14.83 0
 33554432 8388608 float sum -1 3905.6 8.59 15.04 0 3919.0 8.56 14.98 0
 67108864 16777216 float sum -1 7907.1 8.49 14.85 0 7905.3 8.49 14.86 0
 134217728 33554432 float sum -1 16253 8.26 14.45 0 16227 8.27 14.47 0
# Out of bounds values : 0 OK
# Avg bus bandwidth : 4.85265
#

P2P Bandwidth and Latency

Building:

1
2
3


git clone --depth=1 https://github.com/NVIDIA/cuda-samples.git
cd cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest
make -j$(nproc)

Running:

About 21GB/s bidirectional bandwidth
10us+ GPU-GPU latency

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91


# ./p2pBandwidthLatencyTest
P2P Connectivity Matrix
 D\D 0 1 2 3 4 5 6 7
 0 1 0 0 0 0 0 0 0
 1 0 1 0 0 0 0 0 0
 2 0 0 1 0 0 0 0 0
 3 0 0 0 1 0 0 0 0
 4 0 0 0 0 1 0 0 0
 5 0 0 0 0 0 1 0 0
 6 0 0 0 0 0 0 1 0
 7 0 0 0 0 0 0 0 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
 D\D 0 1 2 3 4 5 6 7
 0 909.49 18.77 19.56 20.23 18.72 19.29 19.34 19.83
 1 19.95 919.12 18.93 20.18 18.83 19.23 19.24 19.80
 2 19.87 19.55 919.66 20.27 18.82 19.28 19.29 19.77
 3 19.92 19.48 19.56 918.58 18.82 19.23 19.19 19.80
 4 19.88 20.58 19.74 20.54 920.15 18.38 18.89 18.71
 5 20.04 20.19 19.80 20.37 18.75 921.29 18.88 19.08
 6 20.04 20.11 19.71 20.60 18.77 18.58 920.74 18.62
 7 20.02 20.21 19.73 20.33 18.37 18.74 18.89 921.29
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
 D\D 0 1 2 3 4 5 6 7
 0 910.02 18.83 19.55 20.27 18.76 19.36 19.26 19.83
 1 19.95 920.20 18.92 20.19 18.83 19.26 19.21 19.80
 2 19.82 19.61 919.66 20.28 18.86 19.29 19.34 19.75
 3 19.91 19.47 19.60 920.20 18.83 19.26 19.16 19.81
 4 19.78 20.43 19.75 20.56 920.25 18.60 18.85 18.75
 5 20.02 20.24 19.76 20.45 18.80 922.37 18.89 19.15
 6 20.06 20.12 19.75 20.59 18.80 18.58 921.25 18.61
 7 20.02 20.16 19.80 20.38 18.41 18.72 18.87 920.83
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
 D\D 0 1 2 3 4 5 6 7
 0 916.15 21.03 21.53 21.54 20.69 21.02 21.10 21.06
 1 21.39 920.74 21.59 21.52 20.55 21.02 20.50 21.18
 2 21.15 21.48 921.83 21.20 20.58 21.11 20.98 21.09
 3 21.38 21.67 21.42 922.65 20.53 21.05 20.65 21.21
 4 20.86 20.89 20.86 20.95 923.69 19.99 20.11 19.97
 5 20.90 20.97 21.00 21.02 20.22 922.10 20.15 20.30
 6 20.94 20.62 20.89 20.61 19.98 20.07 923.11 19.99
 7 21.07 21.13 21.10 21.22 20.00 20.29 20.11 923.46
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
 D\D 0 1 2 3 4 5 6 7
 0 917.23 20.86 21.49 21.56 20.63 20.99 21.05 21.13
 1 21.30 922.37 21.61 21.56 20.53 21.08 20.49 21.15
 2 21.15 21.47 922.36 21.16 20.50 21.06 20.94 21.10
 3 21.39 21.60 21.39 922.92 20.63 21.09 20.69 21.22
 4 20.89 20.86 20.81 20.89 923.46 20.12 20.05 20.02
 5 20.88 20.98 20.99 20.87 20.20 923.65 20.12 20.39
 6 21.05 20.63 20.89 20.67 20.07 20.05 922.92 20.02
 7 21.01 21.12 21.06 21.19 20.01 20.31 20.13 923.16
P2P=Disabled Latency Matrix (us)
 GPU 0 1 2 3 4 5 6 7
 0 1.54 20.26 11.68 11.60 12.65 15.80 12.19 11.75
 1 11.60 1.48 11.58 11.66 16.03 12.13 12.55 14.59
 2 20.27 11.59 1.43 20.08 13.17 17.14 16.31 11.41
 3 11.50 11.59 11.51 1.47 13.11 16.14 14.04 11.61
 4 12.53 14.27 12.75 12.65 1.51 12.47 12.49 12.41
 5 11.88 11.73 11.65 11.63 12.28 1.42 14.95 15.40
 6 12.27 12.63 12.42 12.68 12.25 12.30 1.47 11.89
 7 11.46 16.67 11.44 11.47 12.30 13.33 13.57 1.41

 CPU 0 1 2 3 4 5 6 7
 0 4.04 12.23 10.01 9.91 10.88 10.78 10.60 11.29
 1 9.79 3.15 9.85 10.06 10.77 10.56 10.50 10.48
 2 10.04 9.63 3.24 9.94 10.86 10.65 10.70 10.63
 3 9.75 9.61 9.85 3.26 10.92 10.82 10.77 10.67
 4 10.50 10.26 10.61 10.56 3.57 11.55 11.48 11.26
 5 10.52 10.07 10.40 10.48 11.42 3.52 11.32 11.24
 6 10.24 10.20 10.44 10.47 11.42 11.29 3.54 11.35
 7 10.26 10.09 10.50 10.41 11.47 11.31 11.36 3.53
P2P=Enabled Latency (P2P Writes) Matrix (us)
 GPU 0 1 2 3 4 5 6 7
 0 1.54 11.67 11.75 20.27 14.49 15.77 16.95 15.91
 1 11.67 1.48 11.44 11.59 15.22 12.17 13.20 13.18
 2 11.58 20.27 1.43 11.50 14.60 13.73 13.21 11.38
 3 11.58 20.06 12.12 1.47 13.16 16.17 14.67 11.67
 4 12.88 12.78 13.92 12.58 1.50 12.46 12.59 12.50
 5 11.48 11.93 11.78 11.88 19.18 1.42 11.59 11.40
 6 11.95 11.70 14.90 14.00 12.42 11.41 1.46 11.63
 7 12.96 11.45 11.60 11.53 20.55 11.32 12.94 1.40

 CPU 0 1 2 3 4 5 6 7
 0 3.21 9.61 9.86 9.79 10.69 10.48 10.55 10.44
 1 9.67 3.12 9.81 9.81 10.64 10.46 10.66 10.43
 2 9.89 9.62 3.20 9.98 10.96 10.67 10.75 10.72
 3 9.80 9.65 9.89 3.20 10.86 10.78 10.78 10.61
 4 10.41 10.24 10.60 10.50 3.52 11.34 11.49 11.28
 5 10.27 10.14 11.93 14.06 14.53 4.55 16.99 19.67
 6 20.68 20.61 20.39 20.63 20.86 14.35 5.21 14.46
 7 9.89 9.66 9.85 9.85 10.81 10.62 10.68 3.41

How to unlock P2P on RTX4090?

I encountered this legendary modified NVIDIA driver by tinygrad that unlocks P2P on RTX4090 GPUs which blows my mind. Let’s give it a try!

Disable IOMMU

If the following command returns an empty list, it means that the IOMMU is disabled, otherwise, you need to disable it. Consult your motherboard manual on how to disable IOMMU.

1

ll /sys/class/iommu/

Enable Resizable BAR

Enable Resizable BAR in the BIOS. It’s required for the modified NVIDIA driver to work. Consult your motherboard manual on how to enable Resizable BAR.

Uninstall the official NVIDIA driver

We will install our own modified NVIDIA driver, so we need to uninstall the official NVIDIA driver first.

1
2


nvidia-uninstall
reboot

Install the official NVIDIA driver without kernel modules

Visit the modified NVIDIA driver by tinygrad. Check out the branches with p2p in the name. There are multiple versions. You should choose one that fits. For example, I am using the 550.90.07-p2p branch. Remember the version number 550.90.07, you will need it later.

Download the official driver with the same version number as you chose (550.90.07 in my case) from the NVIDIA website. The driver I downloaded is NVIDIA-Linux-x86_64-550.90.07.run.

Install the official driver without the kernel modules because we will install our own modified kernel modules later.

1

./NVIDIA-Linux-x86_64-550.90.07.run --no-kernel-modules

Install the modified kernel modules

Clone the modified NVIDIA driver by tinygrad and install. Remember to replace the branch name with the one you chose.

1
2
3
4
5
6
7
8
9


git clone --depth=1 -b 550.90.07-p2p https://github.com/tinygrad/open-gpu-kernel-modules.git
cd open-gpu-kernel-modules
# Just in case
rmmod nvidia_drm nvidia_modeset nvidia_uvm nvidia
# Build and install the modified kernel modules
make modules -j$(nproc)
make modules_install
depmod
nvidia-smi -pm 1

Test the unlocked performance

PCI BAR

Now the BAR memory is fully available.

256MB -> 32GB.

1
2
3
4
5


# nvidia-smi -q | grep -i bar -A 3
 BAR1 Memory Usage
 Total : 32768 MiB
 Used : 24211 MiB
 Free : 8557 MiB

NCCL Tests

14.47GB/s -> 20.64GB/s: 42% improvement

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


# nThread 1 nGpus 8 minBytes 8 maxBytes 134217728 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
# Rank 0 Group 0 Pid 33895 on <REDACTED> device 0 [0x01] NVIDIA GeForce RTX 4090
# Rank 1 Group 0 Pid 33895 on <REDACTED> device 1 [0x25] NVIDIA GeForce RTX 4090
# Rank 2 Group 0 Pid 33895 on <REDACTED> device 2 [0x41] NVIDIA GeForce RTX 4090
# Rank 3 Group 0 Pid 33895 on <REDACTED> device 3 [0x61] NVIDIA GeForce RTX 4090
# Rank 4 Group 0 Pid 33895 on <REDACTED> device 4 [0x81] NVIDIA GeForce RTX 4090
# Rank 5 Group 0 Pid 33895 on <REDACTED> device 5 [0xa1] NVIDIA GeForce RTX 4090
# Rank 6 Group 0 Pid 33895 on <REDACTED> device 6 [0xc1] NVIDIA GeForce RTX 4090
# Rank 7 Group 0 Pid 33895 on <REDACTED> device 7 [0xe1] NVIDIA GeForce RTX 4090
#
# out-of-place in-place
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
 8 2 float sum -1 38.82 0.00 0.00 0 38.16 0.00 0.00 0
 16 4 float sum -1 38.31 0.00 0.00 0 38.12 0.00 0.00 0
 32 8 float sum -1 38.57 0.00 0.00 0 40.87 0.00 0.00 0
 64 16 float sum -1 40.71 0.00 0.00 0 40.65 0.00 0.00 0
 128 32 float sum -1 40.51 0.00 0.01 0 41.66 0.00 0.01 0
 256 64 float sum -1 41.63 0.01 0.01 0 41.38 0.01 0.01 0
 512 128 float sum -1 41.67 0.01 0.02 0 42.45 0.01 0.02 0
 1024 256 float sum -1 41.70 0.02 0.04 0 41.67 0.02 0.04 0
 2048 512 float sum -1 41.74 0.05 0.09 0 41.74 0.05 0.09 0
 4096 1024 float sum -1 42.35 0.10 0.17 0 42.39 0.10 0.17 0
 8192 2048 float sum -1 42.14 0.19 0.34 0 41.88 0.20 0.34 0
 16384 4096 float sum -1 43.19 0.38 0.66 0 42.73 0.38 0.67 0
 32768 8192 float sum -1 42.17 0.78 1.36 0 42.61 0.77 1.35 0
 65536 16384 float sum -1 47.58 1.38 2.41 0 46.95 1.40 2.44 0
 131072 32768 float sum -1 72.03 1.82 3.18 0 71.50 1.83 3.21 0
 262144 65536 float sum -1 119.7 2.19 3.83 0 118.6 2.21 3.87 0
 524288 131072 float sum -1 166.6 3.15 5.51 0 159.8 3.28 5.74 0
 1048576 262144 float sum -1 216.1 4.85 8.49 0 217.2 4.83 8.45 0
 2097152 524288 float sum -1 321.5 6.52 11.42 0 322.6 6.50 11.37 0
 4194304 1048576 float sum -1 478.7 8.76 15.33 0 480.1 8.74 15.29 0
 8388608 2097152 float sum -1 798.6 10.50 18.38 0 818.0 10.26 17.95 0
 16777216 4194304 float sum -1 1472.8 11.39 19.94 0 1472.1 11.40 19.94 0
 33554432 8388608 float sum -1 2870.3 11.69 20.46 0 2866.4 11.71 20.49 0
 67108864 16777216 float sum -1 5705.0 11.76 20.59 0 5695.6 11.78 20.62 0
 134217728 33554432 float sum -1 11382 11.79 20.64 0 11380 11.79 20.64 0
# Out of bounds values : 0 OK
# Avg bus bandwidth : 6.11168
#

P2P Bandwidth and Latency

P2P is now available: see the 1s in the P2P Connectivity Matrix
Bandwidth: 21.39GB/s -> 50.15GB/s: 134% improvement
Latency: 11.67us -> 1.19us: 89% reduction

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90


P2P Connectivity Matrix
 D\D 0 1 2 3 4 5 6 7
 0 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1
 2 1 1 1 1 1 1 1 1
 3 1 1 1 1 1 1 1 1
 4 1 1 1 1 1 1 1 1
 5 1 1 1 1 1 1 1 1
 6 1 1 1 1 1 1 1 1
 7 1 1 1 1 1 1 1 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
 D\D 0 1 2 3 4 5 6 7
 0 912.13 18.86 19.89 20.25 18.34 19.64 19.25 19.45
 1 20.56 920.74 19.44 20.27 18.46 19.72 19.19 19.40
 2 20.36 19.72 922.37 20.28 18.52 19.65 19.22 19.40
 3 20.43 19.68 20.04 922.49 18.47 19.59 19.15 19.41
 4 20.20 20.15 20.25 20.26 921.83 19.04 18.84 18.36
 5 20.45 20.09 20.17 20.46 18.29 922.37 18.73 18.60
 6 20.39 20.18 20.24 20.54 18.44 19.09 922.49 18.65
 7 20.53 20.14 20.12 20.51 18.17 19.11 18.74 923.46
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
 D\D 0 1 2 3 4 5 6 7
 0 913.74 25.60 25.85 25.85 21.36 22.41 21.74 22.48
 1 25.78 938.91 25.79 25.79 20.70 22.39 22.53 21.63
 2 25.70 25.95 939.00 25.78 21.42 21.59 22.21 22.16
 3 25.60 25.93 25.53 938.42 22.14 22.30 21.70 22.45
 4 22.41 21.61 22.35 22.49 940.70 25.69 25.96 25.90
 5 21.34 22.38 21.72 22.41 25.77 939.57 25.95 25.87
 6 20.57 22.27 22.32 21.60 25.75 26.01 941.73 25.83
 7 21.32 21.52 22.39 22.48 25.51 25.83 25.75 938.44
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
 D\D 0 1 2 3 4 5 6 7
 0 918.85 20.94 21.69 21.67 20.66 21.44 20.98 21.26
 1 21.48 921.29 21.57 21.54 20.20 20.65 20.87 20.70
 2 21.34 21.63 921.56 21.13 20.66 21.28 20.89 21.21
 3 21.40 21.72 21.62 921.83 20.16 20.92 20.89 20.89
 4 20.93 20.33 20.87 20.45 922.90 20.07 20.21 19.96
 5 21.40 20.86 21.26 20.90 20.23 923.69 20.18 20.33
 6 20.93 20.87 20.89 20.90 20.27 20.25 923.67 20.19
 7 21.19 20.66 21.03 20.87 20.02 20.28 20.41 923.41
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
 D\D 0 1 2 3 4 5 6 7
 0 916.45 50.10 50.46 50.29 39.90 41.85 40.83 41.68
 1 50.15 919.60 50.19 50.22 38.93 41.79 41.87 40.82
 2 49.91 50.42 920.74 50.53 40.19 40.94 41.86 41.61
 3 50.08 50.48 50.35 919.66 40.22 41.69 40.92 41.78
 4 41.60 40.94 41.86 41.93 919.93 49.80 50.36 50.22
 5 40.25 41.68 39.96 41.64 49.75 921.83 50.20 50.66
 6 39.59 41.58 41.83 40.95 49.89 50.43 921.56 50.59
 7 40.36 40.86 41.76 41.82 49.96 50.43 50.28 921.01
P2P=Disabled Latency Matrix (us)
 GPU 0 1 2 3 4 5 6 7
 0 1.50 11.60 20.17 20.18 13.08 11.27 14.58 19.33
 1 11.68 1.38 11.74 20.17 12.60 11.39 14.35 14.59
 2 11.33 11.34 1.31 11.50 16.96 20.53 16.53 20.53
 3 11.51 11.51 11.51 1.37 17.86 11.37 15.80 20.54
 4 11.47 11.79 11.44 12.57 1.38 12.24 12.93 11.73
 5 11.48 11.79 11.64 11.64 14.40 1.44 13.46 14.35
 6 13.59 15.05 13.35 12.52 12.06 11.67 1.40 12.34
 7 19.32 19.95 11.39 11.44 12.87 12.23 14.05 1.38

 CPU 0 1 2 3 4 5 6 7
 0 3.28 10.21 10.15 9.83 10.92 10.75 10.88 10.71
 1 10.07 3.14 9.81 9.56 10.58 10.41 10.61 10.35
 2 10.13 9.68 3.28 9.62 10.71 10.56 10.73 10.48
 3 9.81 9.46 9.68 3.13 10.60 10.41 10.52 10.35
 4 10.60 10.20 10.40 10.16 3.48 11.19 11.37 11.07
 5 10.37 10.02 10.29 10.04 11.10 3.49 11.24 11.00
 6 10.51 10.20 10.35 10.11 11.34 11.09 3.47 11.16
 7 10.40 10.11 10.30 10.06 11.17 11.07 11.22 3.42
P2P=Enabled Latency (P2P Writes) Matrix (us)
 GPU 0 1 2 3 4 5 6 7
 0 1.48 1.24 1.26 1.25 1.41 1.41 1.36 1.42
 1 1.19 1.38 1.18 1.18 1.29 1.42 1.29 1.26
 2 1.21 1.20 1.31 1.19 1.30 1.34 1.44 1.41
 3 1.21 1.19 1.17 1.38 1.37 1.41 1.30 1.44
 4 1.69 1.69 1.69 1.64 1.38 1.50 1.59 1.56
 5 1.64 1.60 1.60 1.62 1.50 1.43 1.56 1.48
 6 1.59 1.53 1.57 1.54 1.48 1.44 1.39 1.44
 7 1.58 1.58 1.57 1.56 1.48 1.47 1.45 1.37

 CPU 0 1 2 3 4 5 6 7
 0 3.27 2.84 2.76 2.74 2.80 2.76 2.76 2.75
 1 2.80 3.13 2.66 2.74 2.70 2.70 2.68 2.69
 2 2.83 2.80 3.26 2.85 2.85 2.78 2.77 2.76
 3 2.78 2.69 2.70 3.24 2.69 2.70 2.98 2.67
 4 3.18 3.11 3.10 3.09 3.55 3.12 3.09 3.08
 5 3.12 3.04 3.10 3.05 3.06 3.50 3.08 3.04
 6 3.20 3.11 3.18 3.11 3.11 3.16 3.57 3.08
 7 3.12 3.06 3.08 3.12 3.08 3.07 3.04 3.58

How it works? By tinygrad.

I am not an expert in such low-level hacking (but it’s soooo interesting and I want to learn). This part is from the modified NVIDIA driver by tinygrad (Thank you tinygrad). I just put it here for reference.

Normally, P2P on NVIDIA cards uses MAILBOXP2P. This is some hardware interface designed to allow GPUs to transfer memory back in the days of small BAR. It is not present or disabled in hardware on the 4090s, and that’s why P2P doesn’t work. There was a bug in early versions of the driver that reported that it did work, and it was actually sending stuff on the PCIe bus. However, because the mailbox hardware wasn’t present, these copies wouldn’t go to the right place. You could even crash the system by doing something like torch.zeros(10000,10000).cuda().to("cuda:1")

In some 3090s and all 4090s, NVIDIA added large BAR support.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


tiny@tiny14:~$ lspci -s 01:00.0 -v
01:00.0 VGA compatible controller: NVIDIA Corporation AD102 [GeForce RTX 4090] (rev a1) (prog-if 00 [VGA controller])
 Subsystem: Micro-Star International Co., Ltd. [MSI] Device 510b
 Physical Slot: 49
 Flags: bus master, fast devsel, latency 0, IRQ 377
 Memory at b2000000 (32-bit, non-prefetchable) [size=16M]
 Memory at 28800000000 (64-bit, prefetchable) [size=32G]
 Memory at 28400000000 (64-bit, prefetchable) [size=32M]
 I/O ports at 3000 [size=128]
 Expansion ROM at b3000000 [virtual] [disabled] [size=512K]
 Capabilities: <access denied>
 Kernel driver in use: nvidia
 Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

Notice how BAR1 is size 32G. In H100, they also added support for a PCIe mode that uses the BAR directly instead of the mailboxes, called BAR1P2P. So, what happens if we try to enable that on a 4090?

We do this by bypassing the HAL and calling a bunch of the GH100 methods directly. Methods like kbusEnableStaticBar1Mapping_GH100, which maps the entire VRAM into BAR1. This mostly just works, but we had to disable the use of that region in the MapAperture function for some reason. Shouldn’t matter.

1
2


[ 3491.654009] NVRM: kbusEnableStaticBar1Mapping_GH100: Static bar1 mapped offset 0x0 size 0x5e9200000
[ 3491.793389] NVRM: kbusEnableStaticBar1Mapping_GH100: Static bar1 mapped offset 0x0 size 0x5e9200000

Perfect, we now have the VRAM mapped. However, it’s not that easy to get P2P. When you run ./simpleP2P from cuda-samples, you get this error.

1
2
3
4
5
6
7
8


[ 3742.840689] NVRM: kbusCreateP2PMappingForBar1P2P_GH100: added PCIe BAR1 P2P mapping between GPU2 and GPU3
[ 3742.840762] NVRM: kbusCreateP2PMappingForBar1P2P_GH100: added PCIe BAR1 P2P mapping between GPU3 and GPU2
[ 3742.841089] NVRM: nvAssertFailed: Assertion failed: (shifted >> pField->shift) == value @ field_desc.h:272
[ 3742.841106] NVRM: nvAssertFailed: Assertion failed: (shifted & pField->maskPos) == shifted @ field_desc.h:273
[ 3742.841281] NVRM: nvAssertFailed: Assertion failed: (shifted >> pField->shift) == value @ field_desc.h:272
[ 3742.841292] NVRM: nvAssertFailed: Assertion failed: (shifted & pField->maskPos) == shifted @ field_desc.h:273
[ 3742.865948] NVRM: GPU at PCI:0000:01:00: GPU-49c7a6c9-e3a8-3b48-f0ba-171520d77dd1
[ 3742.865956] NVRM: Xid (PCI:0000:01:00): 31, pid=21804, name=simpleP2P, Ch 00000013, intr 00000000. MMU Fault: ENGINE CE3 HUBCLIENT_CE1 faulted @ 0x7f97_94000000. Fault is of type FAULT_INFO_TYPE_UNSUPPORTED_KIND ACCESS_TYPE_VIRT_WRITE

Failing with an MMU fault. So you dive into this and find that it’s using GMMU_APERTURE_PEER as the mapping type. That doesn’t seem supported in the 4090. So let’s see what types are supported, GMMU_APERTURE_VIDEO,GMMU_APERTURE_SYS_NONCOH, and GMMU_APERTURE_SYS_COH. We don’t care about being coherent with the CPU’s L2 cache, but it does have to go out the PCIe bus, so we rewrite GMMU_APERTURE_PEER to GMMU_APERTURE_SYS_NONCOH. We also no longer set the peer id that was corrupting the page table.

1
2
3
4
5
6
7


cudaMemcpyPeer / cudaMemcpy between GPU0 and GPU1: 24.21GB/s
Preparing host buffer and memcpy to GPU0...
Run kernel on GPU1, taking source data from GPU0 and writing to GPU1...
Run kernel on GPU0, taking source data from GPU1 and writing to GPU0...
Copy data back to host from GPU0 and verify results...
Verification error @ element 1: val = 0.000000, ref = 4.000000
Verification error @ element 2: val = 0.000000, ref = 8.000000

Progress! ./simpleP2P appears to work, however the copy isn’t happening. The address is likely wrong. It turns out they have a separate field for the peer address called fldAddrPeer, we change that to fldAddrSysmem. We also print out the addresses and note that the physical BAR address isn’t being added properly, they provide a field fabricBaseAddress for GMMU_APERTURE_PEER, we reuse it and put the BAR1 base address in there.

Share NVIDIA GPU between CTs in Proxmox VE

Mon, 04 Nov 2024 12:57:00 +0800

Background

Consider a small lab, students need to use GPU for their projects. We have a NVIDIA GPU in our Proxmox VE server, and we want to share the GPU between multiple containers so that multiple students can use the GPU at the same time.

Why not use a VM? Because a GPU can only be passed through to one VM at a time (only one student can use the GPU at a time). And resources are not flexible in VMs.

Why not create multiple users in the host and let them run their programs in the host? Because we want to isolate the students from the host, so that they can’t access the host and other students’ data.

Why not use Docker? Because Docker containers doesn’t have a full init system, and it’s hard to run some applications.

Since we use PVE and it has LXC containers built-in (called CT), it is a perfect choice.

Install Drivers on the Host

Make sure the GPU is detected by the host. Note the NVIDIA GPUs 3b:00.0 (Your address may differ).

1
2
3
4
5


# lspci | grep -i nvidia
3b:00.0 VGA compatible controller: NVIDIA Corporation TU104GL [Quadro RTX 5000] (rev a1)
3b:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)
3b:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1)
3b:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)

You may ask: does your entire lab only own one RTX 5000? What kind of lab is this? Are you cave people?

Yes, although we have multiple projects worth over millions of Chinese Yuan, most of the money is gone to the some other places (which I cannot publicly speak on the Internet 🤫 ). And the professors have no emphasis on students’ growth. As a result, we are actually poor as hell.

Since almost no one knows how to properly configure a Linux server, I want to help my classmates to learn more and let them use the only GPU. But to be honest, I won’t benefit from doing this. It’s just voluntary work.

Install prerequisites. Note that I am using pve-headers-$(uname -r) to install the headers for the current kernel. If you are using a different kernel, you may need to install the headers for that kernel. Also, you may want to use linux-headers-$(uname -r) instead of pve-headers-$(uname -r) if you are not using Proxmox VE.

1

# apt install -y gcc make pve-headers-$(uname -r)

Download CUDA toolkit from here and install it. Drivers are included in the CUDA toolkit so you don’t need to install drivers separately.

1
2


# wget <cuda-runfile-download-url>
# ./cuda_12.2.2_535.104.05_linux.run --silent

The default installation options will work fine. If anything fails, you can check the log file at /var/log/cuda-installer.log for CUDA logs and /var/log/nvidia-installer.log for NVIDIA driver logs.

PS: You need to blacklist nouveau driver. This is automatically done by PVE. If not, you can do this by creating a file /etc/modprobe.d/blacklist-nouveau.conf with the following content: blacklist nouveau. Then run update-initramfs -u to update the initramfs.

PPS: If you used to passthrough this GPU to a VM, be sure to remove the GPU from the VM’s hardware configuration in PVE otherwise PVE will bound the GPU to vfio-pci (see Kernel driver in use row in lspci -k) and cannot be used by the host.

PPPS: Some kernel versions are known to have problems with NVIDIA drivers. If you encounter problems, you may need to downgrade/upgrade the kernel. For example, kernel version 5.10.0 is known to have make[3]: *** No rule to make target 'scripts/module.lds', needed by '/tmp/selfgz38416/NVIDIA-Linux-x86_64-560.35.03/kernel-open/nvidia.ko' error.

After installation finished, check if the driver is loaded.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# nvidia-smi
Tue Nov 5 09:56:44 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Quadro RTX 5000 Off | 00000000:3B:00.0 Off | Off |
| 33% 44C P0 28W / 230W | 0MiB / 16384MiB | 6% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+

Allow NVIDIA Device Passthrough in CT

Now we need to allow the CT to access the GPU. I am using an unprivileged container here. Edit the CT’s configuration file (/etc/pve/local/lxc/<id>.conf). Add the following lines to the end of the file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


 arch: amd64
 cores: 4
 features: nesting=1
 hostname: ct-gpu-tmpl-deb127-cu122
 memory: 4096
 net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=AA:AB:F0:07:42:D0,ip=dhcp,type=veth
 ostype: debian
 rootfs: local-zfs:basevol-8001-disk-0,size=16G
 swap: 0
 unprivileged: 1
# These lines allow the container to access specific character devices (c) with rwm
# permissions (read, write, modify). These are needed for NVIDIA GPU access.
+ lxc.cgroup.devices.allow: c 195:* rwm
+ lxc.cgroup.devices.allow: c 509:* rwm
+ lxc.cgroup.devices.allow: c 235:* rwm
# These lines mount various GPU-related devices from the host into the container.
+ lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
+ lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
+ lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
+ lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
+ lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
+ lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
+ lxc.mount.entry: /dev/fb0 dev/fb0 none bind,optional,create=file

PS: If you cannot use nvidia-smi (it says Failed to initialize NVML: Unknown Error), there is a possibility that you are using cgroup2. Change all the lxc.cgroup.devices.allow lines to lxc.cgroup2.devices.allow.

Explanation:

Allows container access to NVIDIA device nodes:

c 195:* - NVIDIA character devices
c 509:* - NVIDIA UVM devices
c 235:* - NVIDIA CTL devices

Maps the following host GPU devices into container:

/dev/nvidia0 - Main GPU device
/dev/nvidiactl - NVIDIA control device
/dev/nvidia-modeset - Display mode setting
/dev/nvidia-uvm - Unified memory management
/dev/nvidia-uvm-tools - UVM diagnostic tools
/dev/dri - Direct Rendering Infrastructure
/dev/fb0 - Framebuffer device

Mount options:

bind: Mount as a bind mount
optional: Don’t fail if device doesn’t exist
create=file/dir: Create the mount point if it doesn’t exist

Note that if you are using a different GPU, you may need to change the device numbers. For example, /dev/nvidia1 instead of /dev/nvidia0. You can find the device numbers in nvidia-smi output.

Install Drivers in CT

Log into the CT. All the following commands are run in the CT.

You should be able to see NVIDIA devices inside the CT:

1
2
3
4
5
6


# ls -l /dev/nvidia*
---------- 1 root root 0 Nov 5 02:31 /dev/nvidia-modeset
crw-rw-rw- 1 nobody nogroup 507, 0 Nov 5 01:56 /dev/nvidia-uvm
crw-rw-rw- 1 nobody nogroup 507, 1 Nov 5 01:56 /dev/nvidia-uvm-tools
crw-rw-rw- 1 nobody nogroup 195, 0 Nov 5 01:56 /dev/nvidia0
crw-rw-rw- 1 nobody nogroup 195, 255 Nov 5 01:56 /dev/nvidiactl

Install CUDA and drivers, just like you would on a physical machine, except that you don’t need to install the kernel modules. I will install CUDA 12.2 (drivers are included in the CUDA installer).

1
2
3


# wget <cuda-runfile-download-url>
# apt install -y gcc
# ./cuda_12.2.2_535.104.05_linux.run --extract=$(pwd)/cu122

Note that I extracted the installer to manually install it because we want to skip kernel module installation and such options are not exposed in the installer.

Install the bundled drivers:

1
2


cd cu122
./NVIDIA-Linux-x86_64-535.104.05.run --no-nouveau-check --no-kernel-modules --silent

Run nvidia-smi to check if the driver is loaded.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# nvidia-smi
Tue Nov 5 05:49:02 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Quadro RTX 5000 Off | 00000000:3B:00.0 Off | Off |
| 33% 38C P0 23W / 230W | 0MiB / 16384MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+

We can now see that the GPU is accessible in the CT.

Let’s continue with the CUDA installation. Remember to uncheck the driver installation option because we have already installed the drivers above.

1

./cuda-linux.12.2.2-535.104.05.run

After a successful installation, you should add cuda binaries to PATH. Instructions should be printed at the end of the installation. Then you can run nvcc to see if CUDA is installed correctly.

Everything should be working by this point.

Missing `nvidia-uvm` and High Idle Power Draw

One problem I encountered is that when the host reboots, the GPU is not accessible in the CT. This is because nvidia-uvm device isn’t created until an application attempts to interact with the graphics card. This is a problem because no application will interact with the GPU at boot, so no nvidia-uvm device is created. But the CT needs the nvidia-uvm device bind-mounted at CT-startup in order to access the GPU.

Also, the graphics card have insanely high power draw at idle (over 100 Watts). The GPU is in P0 and never leaves it. We can use nvidia-persistenced to let the GPU enter a low-power state (P8) when not in use.

To solve this, we can run nvidia-smi -pm 1 (which enables nvidia-persistenced and keeps nvidia character device and handles frequency scaling) at boot. Add the following line to the host’s crontab to run nvidia-smi -pm 1 at boot.

PS: This only works if the host is a headless server (no monitor attached). If you have a monitor attached, you may need to run nvidia-smi below instead.

1
2


# crontab -e
@reboot /usr/bin/nvidia-smi -pm 1

Downsides

Despite the fact that this method works best for us, there are some downsides:

The CT will have full access to the GPU. If one CT uses all the GPU memory, other CTs will be starving. So you must trust the users of the CTs. This is not a problem for us because we know each other.
Driver updates are a bit more complicated. You need to update the drivers on the host and in all of the CTs. It’s best to not update the drivers too often.
The CTs share the same kernel with the host. To avoid potential compatibility issues, we don’t update the kernel unless necessary.

GPU on Charlie Chiang's blog

Proxmox VE Shared GPU Installation Guide

Introduction

Proxmox VE Installation

Proxmox VE Post-Installtion

Change APT Sources

Change CT Template Sources

Stop Cluster Services

Install Common Tools

Configure Shell

Configure ZRAM Swap

Enable IPv6 SLAAC

TRIM Optimizations

Limit Journal Size

NVIDIA P2P Driver Installation

Disable IOMMU

Disable ACS

Enable Above 4G Decoding

Install Official Driver

Install P2P-Enabled Kernel Modules

Enable Persistence Mode

Build CT Templates

Install Docker

Revert Containerd Config

Add GPUs to CT

Install GPU Driver

Install CUDA

Install NCCL

Install NVIDIA Container Toolkit

Install NVIDIA Nsight Systems

Functionality Tests

NCCL

CUDA P2P

Prepare for Templating

Other Minor Settings

RAID Card Write Back

Obstacles

Only 255 Cores are Recognized - x2APIC

RTX4090 GPUDirect P2P Unlocked!

What is P2P?

Why is P2P disabled on RTX4090?

Test machine configuration

Test the original performance

PCI BAR

NCCL Tests

P2P Bandwidth and Latency

How to unlock P2P on RTX4090?

Disable IOMMU

Enable Resizable BAR

Uninstall the official NVIDIA driver

Install the official NVIDIA driver without kernel modules

Install the modified kernel modules

Test the unlocked performance

PCI BAR

NCCL Tests

P2P Bandwidth and Latency

How it works? By tinygrad.

Share NVIDIA GPU between CTs in Proxmox VE

Background

Install Drivers on the Host

Allow NVIDIA Device Passthrough in CT

Install Drivers in CT

Missing nvidia-uvm and High Idle Power Draw

Downsides

Missing `nvidia-uvm` and High Idle Power Draw