Proxmox VE on Charlie Chiang's blog

Installing Proxmox VE on an OVHcloud Kimsufi KS-5

Mon, 06 Jul 2026 14:30:00 +0800

I ordered an OVHcloud Kimsufi KS-5 dedicated server for 19.90 USD/month. It is not fast by modern standards, but it is a very usable small Proxmox box if you install it carefully.

The goal for this machine was:

Proxmox VE on mirrored ZFS root.
Keep only part of each SSD for the root pool.
Use the remaining SSD space as a fast disposable scratch pool.
Put VMs and CTs behind NAT, because the server only has one usable public IPv4 address and one /128 IPv6 address.
Keep host writes low where it is easy to do so.

This post is written from the actual KS-5 I installed. The finished machine is running Proxmox VE 9.2.

Hardware

The server I received:

Part	Value
Product	Kimsufi KS-5
CPU	Intel Xeon E3-1270 v6, 4C/8T, 3.8 GHz base, 4.2 GHz turbo
RAM	32 GB DDR4 ECC 2400 MHz
Storage	2x Intel P3520 450 GB NVMe SSD
NIC	2x Intel I210 gigabit

The PCI devices looked like this:

1
2
3
4
5
6
7
8


00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers (rev 05)
00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31)
00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)
02:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 02)
03:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 02)
04:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1) (rev 05)
05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

The SSDs were not new in power-on hours, but the wear was low and both drives had no media errors:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# nvme smart-log /dev/nvme0n1
critical_warning : 0
temperature : 23 C
available_spare : 98%
percentage_used : 6%
Data Units Read : 8.41 TB
Data Units Written : 95.99 TB
power_cycles : 107
power_on_hours : 45628
unsafe_shutdowns : 4
media_errors : 0
num_err_log_entries : 0

# nvme smart-log /dev/nvme1n1
critical_warning : 0
temperature : 23 C
available_spare : 98%
percentage_used : 7%
Data Units Read : 142.27 TB
Data Units Written : 112.67 TB
power_cycles : 74
power_on_hours : 54153
unsafe_shutdowns : 5
media_errors : 0
num_err_log_entries : 0

For a cheap dedicated server, that is acceptable.

Why Install Proxmox Manually

OVHcloud can install an OS for you from the control panel. I did not use that path for Proxmox.

The problem is storage layout. Proxmox works best when it owns the disks directly and can put ZFS on the raw devices. The OVHcloud installer tends to build layouts around mdadm RAID and regular filesystems, or it uses ZFS in a way that treats VM/CT storage more like ordinary directories. That leaves some Proxmox storage features unavailable or awkward, especially thin-provisioned disks and the normal ZFS-backed workflow.

So I installed Proxmox myself.

Format NVMe Drives as 4K LBA

Before installing the OS, boot the server into OVHcloud rescue mode and check the NVMe namespace formats.

My Intel P3520 drives supported both 512-byte and 4096-byte LBA formats:

1
2
3
4
5
6
7
8


# nvme id-ns -H /dev/nvme0n1
LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
LBA Format 1 : Metadata Size: 8 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good
LBA Format 2 : Metadata Size: 16 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good
LBA Format 3 : Metadata Size: 0 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best
LBA Format 4 : Metadata Size: 8 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best
LBA Format 5 : Metadata Size: 64 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best
LBA Format 6 : Metadata Size: 128 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best

The drives defaulted to 512-byte LBA. Since these are enterprise SSDs and 4K was available, I reformatted both namespaces to format 3: 4096-byte sectors with no metadata.

This erases all data on the drive.

1
2


nvme format /dev/nvme0n1 -l 3
nvme format /dev/nvme1n1 -l 3

After formatting, nvme list should show 4 KiB + 0 B:

1
2
3


Node Model Namespace Usage Format FW Rev
/dev/nvme0n1 INTEL SSDPE2MX450G7 0x1 450.10 GB / 450.10 GB 4 KiB + 0 B MDV10290
/dev/nvme1n1 INTEL SSDPE2MX450G7 0x1 450.10 GB / 450.10 GB 4 KiB + 0 B MDV10290

Open the IPMI KVM

In the OVHcloud console, open the IPMI KVM.

The KVM launches through an old Java JNLP applet.

One annoying detail: the IP address you use to access the OVHcloud console should be the same public client IP used when opening the IPMI KVM. I hit a failure mode where the KVM was blocked because I accessed them through different egress IPs.

The machine uses an Intel server board. The useful hotkeys are:

F2: BIOS setup.
F6: one-time boot menu.

The boot logo confirms it is an Intel board:

You can mount a virtual ISO from the JViewer client:

Then use F6 and choose the virtual CD-ROM:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


Please select boot device:

UEFI IPv4: Intel I210 Network 00 at Baseboard
UEFI IPv4: Intel I210 Network 00 at Baseboard 2
UEFI IPv6: Intel I210 Network 00 at Baseboard
UEFI IPv6: Intel I210 Network 00 at Baseboard 2
Launch EFI Shell
Enter Setup
UEFI Virtual CDROM 1.00 # Choose this one!
UEFI Misc Device

Do Not Stream the Proxmox ISO Through JViewer

My first attempt was to mount the Proxmox installer ISO directly in JViewer and boot it.

That technically works, but it was unusably slow. Even when I ran JViewer from an OVH server (which should have enough bandwidth), virtual media throughput was capped at around 64 KB/s. Installing Proxmox by streaming a full ISO through that path would take forever.

The fix is to boot a tiny netboot image first.

Boot Proxmox Through netboot.xyz

Download the netboot.xyz UEFI ISO, mount that ISO in JViewer, and boot it. The image is tiny, so the slow virtual media path is no longer a problem.

In netboot.xyz, choose:

1
2
3


Linux Network Installs (64-bit)
Proxmox
Proxmox VE Text Installer

You can choose the debug installer if you want shell access between install stages. That is useful when something goes wrong.

netboot.xyz downloads the Proxmox installer over the server’s own network connection and then boots it. OVHcloud provides DHCP even for dedicated servers, so the installer had network access without manual IP configuration.

One caveat: after netboot.xyz hands off to the Proxmox installer, the installer does not use a serial console, so Serial over LAN will not work. You still need the JViewer window to see and control the installation.

Install Proxmox on ZFS RAID1

In the Proxmox installer, choose ZFS RAID1 across both NVMe drives.

I intentionally did not give the full SSDs to rpool. The installer has an hdsize option in advanced storage options. I set it to 128 GiB so the root pool would be a 2-way mirror, leaving the remaining space on both SSDs unused for a later scratch pool.

This is the important part: decide this during installation. Growing ZFS into extra space is easy. Shrinking an existing ZFS pool is not.

The final root layout on my machine:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


nvme0n1 419.2G disk
|-nvme0n1p1 1000K part
|-nvme0n1p2 1G part vfat
|-nvme0n1p3 128G part zfs_member
`-nvme0n1p4 290.2G part zfs_member

nvme1n1 419.2G disk
|-nvme1n1p1 1000K part
|-nvme1n1p2 1G part vfat
|-nvme1n1p3 128G part zfs_member
`-nvme1n1p4 290.2G part zfs_member

p3 on both drives is the mirrored root pool:

1
2
3
4
5
6
7
8
9


pool: rpool
state: ONLINE
config:

 NAME STATE
 rpool ONLINE
 mirror-0 ONLINE
 nvme-INTEL_SSDPE2MX450G7_CVPF71620037450RGN-part3 ONLINE
 nvme-INTEL_SSDPE2MX450G7_CVPF721600N5450RGN-part3 ONLINE

p4 on both drives is the later scratch pool.

Optional Partition Alignment Fix

The Proxmox installer did not give me a perfectly round 1 MiB-aligned final size for the ZFS partition (nvmeXn1p3). If you care about this, fix it immediately after installation while the layout is still simple.

The safe method is:

Use fdisk /dev/nvmeXn1
List partitions with p and note the start/end sector of part 3.
Delete d part 3
Create a new part 3 with the same start sector and a new end sector. Increase the end sector a bit, align end sector - 1 to 1M (256 4k sectors).
Do not wipe the ZFS signature.
Write changes with w.
Ask ZFS to expand into the recreated partition.

Example:

1

zpool online -e rpool <device>

Note that if you expanded the ZFS part too little (smaller than metaslabs size, typically 1GiB), ZFS may not actually expand the pool to use the empty space. This is fine.

Find the exact <device> with:

1

zpool status -v rpool

This is not required for a working system. I did it because I wanted the 128 GiB root partition to be exact and aligned. So later partition 4 will not be misaligned either.

Fix OVHcloud Boot-to-Disk

After a manual install, the server may not boot straight into Proxmox even though the installation succeeded.

OVHcloud bare-metal boot is not just “BIOS loads local disk”. The normal path is roughly:

The server PXE-boots from the public interface.
OVHcloud DHCP gives the server its public IP and an iPXE loader.
iPXE queries OVHcloud’s internal boot service.
The boot service returns a script based on your configured boot mode.
In boot-to-disk mode, iPXE uses sanboot with an EFI bootloader path.

If you installed through the OVHcloud panel, OVHcloud knows the EFI bootloader path. If you installed manually, that path can be missing or wrong.

When sanboot fails, iPXE falls back to rEFInd. rEFInd scans the EFI system partitions and may choose the wrong bootloader. In my case it picked memtest first:

1
2
3
4


rEFInd - Booting OS

Starting memtest86+x64.efi
Using load options ''

The rEFInd menu showed both memtest and systemd-boot:

1
2
3
4


Boot memtest86+x64.efi from 1021 MiB FAT volume
Boot EFI\systemd\systemd-bootx64.efi from 1021 MiB FAT volume
Boot memtest86+x64.efi from 1021 MiB FAT volume
Boot EFI\systemd\systemd-bootx64.efi from 1021 MiB FAT volume

For Proxmox installed with UEFI and root-on-ZFS, the bootloader is systemd-boot:

1

\EFI\systemd\systemd-bootx64.efi

Set that path through the OVHcloud CLI:

1
2


ovhcloud login
ovhcloud baremetal edit nsXXX.ip-X-X-X.eu --efi-bootloader-path '\EFI\systemd\systemd-bootx64.efi'

After that, boot-to-disk should go directly into Proxmox instead of falling through to rEFInd.

Basic Post-Install ZFS Settings

Enable TRIM on the root pool and set basic dataset properties:

1
2
3
4
5
6


zpool set autotrim=on rpool
zfs set atime=off rpool
# The default 'on' is also lz4. Just set it to lz4 explicitly to be sure.
zfs set compression=lz4 rpool
# 128k is the default (this is good for HDDs). But 16-32k is better for NVMe array and workloads with small files, which is the case for most VMs/CTs and Proxmox OS. I am using 2-way mirror so I don't need to worry about 4k ashift adding up.
zfs set recordsize=16k rpool

recordsize affects file datasets, not zvol block devices. For VM disks backed by ZFS zvols, check the Proxmox storage block size instead. On new Proxmox installations the default is already 16K; older installations may still use 8K. You can check or change it in:

1

Datacenter -> Storage -> local-zfs -> Edit -> Block Size

I also use a few ZFS module options:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# You may remove zfs_arc_max set by the installer if you want to maximize ZFS ARC. Note that ARC may not shrink fast enough under memory pressure, so keep this in mind.

cat >/etc/modprobe.d/zfs.conf <<'EOF'
options zfs zfs_txg_timeout=30
options zfs zfs_trim_txg_batch=128
options zfs zfs_dirty_data_sync_percent=80
options zfs zfs_delay_min_dirty_percent=95
EOF

update-initramfs -u -k all

The intent:

zfs_txg_timeout=30: reduce idle write frequency.
zfs_trim_txg_batch=128: make TRIM batching less tiny for NVMe.
zfs_dirty_data_sync_percent=80 and zfs_delay_min_dirty_percent=95: delay throttling until the dirty-data situation is actually serious.

These are not universal defaults. They fit this small host because the workload is mostly personal VMs/CTs and scratch tasks, not a database with strict latency guarantees.

Reduce Unnecessary Host Writes

I am not using Proxmox clustering on this server, so I disabled the cluster HA services:

1
2
3


systemctl disable --now pve-ha-crm.service
systemctl disable --now pve-ha-lrm.service
systemctl disable --now corosync.service

Limit persistent journald usage:

1
2


sed -i 's/.*SystemMaxUse.*/SystemMaxUse=128M/g' /etc/systemd/journald.conf
systemctl restart systemd-journald

Configure BBR

Use BBR with fq.

On Debian/Proxmox, a later default sysctl file can set net.core.default_qdisc=fq_codel after your file during boot. Use a late filename:

1
2
3
4
5
6


cat >/etc/sysctl.d/99-bbr.conf <<'EOF'
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
EOF

sysctl --system

Check it:

1

sysctl net.core.default_qdisc net.ipv4.tcp_congestion_control

Expected:

1
2


net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

Configure Swap

Always have swap when using ZFS, even with enough RAM. It gives the kernel somewhere to go if ARC does not shrink quickly enough under memory pressure, and it reduces the chance that the OOM killer targets useful applications.

I use ZRAM swap:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


git clone --depth=1 https://github.com/foundObjects/zram-swap.git
cd zram-swap
./install.sh
cd ..
rm -rf zram-swap

cat >/etc/sysctl.d/zram.conf <<'EOF'
vm.swappiness = 180
vm.watermark_boost_factor = 0
vm.watermark_scale_factor = 125
vm.page-cluster = 0
EOF

sysctl --system

You may use disk swap if you want.

My current host has a 38.8 GiB ZRAM swap device:

1
2


NAME TYPE SIZE USED PRIO
/dev/zram0 partition 38.8G 0B 15

SSH and Serial Console

I like SSH sessions to die quickly when the client disappears unexpectedly:

1
2


ClientAliveInterval 60
ClientAliveCountMax 3

Put that in sshd_config or a file under sshd_config.d, then restart SSH.

For emergency access, enable OVHcloud serial-over-LAN by adding a serial console to the Proxmox kernel command line:

1
2
3


additional_cmdline="console=tty0 console=ttyS0,115200n8"
sed -i "s|$| $additional_cmdline|" /etc/kernel/cmdline
proxmox-boot-tool refresh

On my host, /etc/kernel/cmdline now contains:

1

root=ZFS=rpool/ROOT/pve-1 boot=zfs vmlinuz video=vesafb:ywrap,mtrr initrd=initrd.magic console=tty0 console=ttyS0,115200n8

Scratch Storage

The root pool uses 128 GiB on each SSD. The remaining 290.2 GiB on each SSD is free for scratch storage.

I considered two options:

mdadm RAID0 + LVM thin + ext4/XFS.
ZFS striped pool.

The mdadm option may win in some microbenchmarks, but the management cost is higher. I used a ZFS striped pool because it is simple and good enough.

This pool has no redundancy. If either SSD fails, the scratch pool is gone. That is fine for my use case because it is scratch space.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


zpool create -f \
 -o ashift=12 \
 -o autotrim=on \
 scratchpool \
 /dev/disk/by-id/nvme-INTEL_SSDPE2MX450G7_CVPF721600N5450RGN-part4 \
 /dev/disk/by-id/nvme-INTEL_SSDPE2MX450G7_CVPF71620037450RGN-part4

zfs set recordsize=16k scratchpool
zfs set atime=off scratchpool
zfs set compression=lz4 scratchpool
zfs set logbias=throughput scratchpool

zfs create scratchpool/unsafe
zfs set sync=disabled scratchpool/unsafe
zfs set copies=1 scratchpool/unsafe

Current state:

1
2
3
4
5
6
7
8


pool: scratchpool
state: ONLINE
config:

 NAME STATE
 scratchpool ONLINE
 nvme-INTEL_SSDPE2MX450G7_CVPF721600N5450RGN-part4 ONLINE
 nvme-INTEL_SSDPE2MX450G7_CVPF71620037450RGN-part4 ONLINE

Properties:

1
2
3


NAME AVAIL MOUNTPOINT RECSIZE COMPRESS ATIME SYNC COPIES
scratchpool 562G /scratchpool 16K lz4 off standard 1
scratchpool/unsafe 562G /scratchpool/unsafe 16K lz4 off disabled 1

Use scratchpool/unsafe only for data you can recreate. sync=disabled lies to applications about sync writes.

mdadm + LVM Alternative

If you want the non-ZFS scratch layout, create one extra partition on each SSD and build RAID0:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


apt install -y mdadm

mdadm --create /dev/md0 \
 --verbose \
 --level=0 \
 --raid-devices=2 \
 --chunk=128K \
 /dev/disk/by-id/nvme-INTEL_SSDPE2MX450G7_CVPF721600N5450RGN-part4 \
 /dev/disk/by-id/nvme-INTEL_SSDPE2MX450G7_CVPF71620037450RGN-part4

mdadm --detail --scan >> /etc/mdadm/mdadm.conf
update-initramfs -u

For alignment:

mdadm chunk size: 128K. 128K is a good balance between performance and overhead for NVMe drives. If you want to maximize sequential performance, use larger values like 1M. But remember to align filesystem / LVM with this value.
Data disks: 2.
Stripe width: 256K.
LVM data alignment: 1M is fine because it is a multiple of 256K.

NVMe (physical) → mdadm RAID0 → LVM PV → LVM LV / thin pool → filesystem

When you create LVM on top of RAID-0, you must ensure:

LVM PE size = multiple of mdadm chunk size
LVM PV starts aligned to mdadm stripe boundary

Create the LVM layer:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# The default dataalignment value is 1M, which is a multiple of 256K, so it's fine.
pvcreate --dataalignment 1M /dev/md0
# Default VG extent size is often 4M, which is multiples of 1M data alignment, 
# which is also multiples of 256K stripe width. Using the default value is fine.
vgcreate --physicalextentsize 4M vg_scratch /dev/md0
# Thin pool chunk size defines how much physical space is reserved when any 
# part of a chunk is touched. It does not defined write size. So it can be 
# smaller than 256k. The default 64k is fine. I will just match it with the 
# stripe width = 256k.
# The default metadata size can be too big if you are using small driver 
# like 16G Optane drive. Calculate the desired metadata size (bytes)
# by `48 * (total size / chunk size)`.
lvcreate -l 100%FREE --thin vg_scratch/scratch_thin --chunksize 256k --poolmetadatasize 512M

When formatting a filesystem inside a VM or manually on the host, pass the stripe geometry:

1
2
3
4
5
6
7
8
9


# ext4
# stride = mdadm chunk size / filesystem block size = 128k / 4k = 32
# stripe-width = stride * disks = 32 * 2 = 64
mkfs.ext4 -b 4096 -E stride=32,stripe-width=64 /dev/vg_scratch/lv

# XFS
# su = mdadm chunk size = 128k
# sw = data disks = 2
mkfs.xfs -d su=128k,sw=2 /dev/vg_scratch/lv

Fast but unsafe mount options for scratch data:

1
2


ext4: noatime,nodiratime,nobarrier,data=writeback,commit=60
xfs: noatime,nodiratime,logbufs=8,logbsize=256k

For CTs, Proxmox does not make it convenient to set filesystem mount options on a normal CT mountpoint. If you need custom options, create the LV manually, mount it on the host, and bind-mount it into the CT:

1
2
3
4
5
6
7
8


lvcreate -V 16G -T vg_scratch/scratch_thin -n scratch_test
mkfs.xfs -d su=128k,sw=2 /dev/vg_scratch/scratch_test

mkdir -p /mnt/scratch_test
mount -o noatime,nodiratime,logbufs=8,logbsize=256k /dev/vg_scratch/scratch_test /mnt/scratch_test
chmod 777 /mnt/scratch_test

pct set 199 -mp0 /mnt/scratch_test,mp=/mnt/scratch_test

If your workload is random-write heavy and can fit on one SSD, skip RAID0 and LVM entirely. A single raw disk or single partition is often better for small random writes than a layered RAID0 + LVM-thin setup.

Using Scratch Storage in VMs and CTs

For VMs, add a virtual disk on the scratch pool. Enable discard and SSD emulation on the virtual disk. Inside the guest, enable periodic TRIM. When formatting the disk, make sure the filesystem is aligned to the underlying ZFS volsize or LVM stripe geometry. For example, if underlying ZFS volsize is 16K, use mkfs.ext4 -O bigalloc -C 16384 /dev/yourdisk. If the underlying storage is LVM+RAID, use the correct stride and stripe-width options when formatting.

For CTs, add a mountpoint on the scratch pool. ZFS mountpoints are automatically aligned. For LVM+RAID, make sure the filesystem is aligned to the underlying geometry. TRIM is handled by the host for ZFS mountpoints. For LVM+RAID, run pct fstrim on the CT.

To use the scratch pool for temporary data, you can use bind-mounts to mount common scratch directories to scratch disk mountpoints. This avoids changing the config for each applications such as Docker, containerd and etc.

Suppose the scratch disk is mounted at /mnt/unsafe, and you want to use it for Docker, containerd, and ~/.cache. Write the following lines in /etc/fstab:

1
2
3


/mnt/unsafe/docker /var/lib/docker none rbind 0 0
/mnt/unsafe/containerd /var/lib/containerd none rbind 0 0
/mnt/unsafe/.cache /root/.cache none rbind 0 0

1
2
3
4


mkdir -p /mnt/unsafe/{docker,containerd,.cache}
rm -rf /var/lib/docker /var/lib/containerd /root/.cache
mkdir -p /var/lib/docker /var/lib/containerd /root/.cache
mount -av

TRIM

For ZFS pools:

1
2


zpool set autotrim=on rpool
zpool set autotrim=on scratchpool

For VMs, enable discard and SSD emulation on virtual disks, then enable periodic TRIM inside the guest.

For CTs backed by ZFS mountpoints, the host handles it. For CTs backed by other storage such as LVM thin, run pct fstrim:

1
2
3
4
5
6
7
8
9


cat >/usr/local/bin/trimcts.sh <<'EOF'
#!/bin/bash
for id in $(pct list | awk '$2 == "running" {print $1}'); do
 echo "==> Trimming CT $id"
 pct fstrim "$id"
done
EOF

chmod +x /usr/local/bin/trimcts.sh

Then run it from cron or a systemd timer. For example, weekly from cron:

1

0 2 * * 7 /usr/local/bin/trimcts.sh >/var/log/trimcts.log 2>&1

Proxmox Networking on a Single Public IP

Proxmox creates a bridge by default and attaches the physical NIC to it. That is usually the right setup when the server is in your own LAN.

It is not right for this Kimsufi server.

The server has one usable public IPv4 address and one /128 IPv6 address. It does not have a routed public subnet for VMs and CTs. If VMs are bridged directly to the public interface, they do not get usable public addresses.

So I do this instead:

Put the public IPv4 and public IPv6 directly on eno1, not vmbr0. By default, Proxmox puts the public IP on the vmbr0 bridge, which is not what we want. You should move it to the physical interface.
Create vmbr0 as an internal bridge with no physical ports.
Put VMs and CTs on vmbr0.
NAT outbound traffic from vmbr0 to eno1.
Port-forward selected inbound ports to service CTs.

Current /etc/network/interfaces:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
 address <public-ipv4>/24
 gateway <public-ipv4-gateway>

iface eno1 inet6 static
 address <public-ipv6>/128
 gateway <public-ipv6-gateway>

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
 address 10.187.54.1/24
 bridge-ports none
 bridge-stp off
 bridge-fd 0

iface vmbr0 inet6 static
 address fc10:187:54::1/64

source /etc/network/interfaces.d/*

Enable forwarding:

1
2
3
4
5
6


cat >/etc/sysctl.d/99-forward.conf <<'EOF'
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF

sysctl --system

About NAT66: it is usually the wrong IPv6 design. If you have a routed IPv6 prefix, route it to your VM bridge. On this server I only have one /128, so NAT66 is the practical workaround if VMs/CTs need outbound IPv6.

NAT and Port Forwarding

NAT rewrites packet addresses as traffic crosses the host.

For outbound IPv4 NAT, a CT sends a packet like:

1

src=10.187.54.100:51514 dst=1.1.1.1:443

After MASQUERADE on the host:

1

src=<public-ipv4>:51514 dst=1.1.1.1:443

Linux conntrack records that translation so the reply can be translated back and forwarded to the CT.

There is a Proxmox-specific caveat when the firewall is enabled on a VM or CT network interface (firewall=1). Proxmox places that interface behind an fwbr* firewall bridge. With bridge netfilter enabled, conntrack can record the connection while it crosses the firewall bridge, before it reaches the host’s normal NAT path. The later FORWARD rule can accept the packet, but POSTROUTING may reuse the already tracked non-NATed connection instead of attaching MASQUERADE or SNAT. The packet then leaves the public interface with its private source address and never receives a reply.

Put the firewall-bridge tracking in a separate conntrack zone by adding these hooks to the vmbr0 stanza in /etc/network/interfaces:

1
2
3
4


 post-up iptables -t raw -C PREROUTING -i fwbr+ -j CT --zone 1 2>/dev/null || iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1 2>/dev/null || true
 post-up ip6tables -t raw -C PREROUTING -i fwbr+ -j CT --zone 1 2>/dev/null || ip6tables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
 post-down ip6tables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1 2>/dev/null || true

The -C/|| guard matters on a dual-stack vmbr0: ifupdown2 can run the hook once for each address family.

The IPv4 rules cover MASQUERADE/SNAT, while the IPv6 rules are also needed when using NAT66. A characteristic symptom is that NAT works for a guest without the Proxmox interface firewall but fails for an otherwise identical firewall-enabled guest. In conntrack output, the broken flow’s reply destination remains the guest’s private address instead of the host’s public address. Allowing all OUT and FORWARD traffic does not fix it because this is a conntrack/NAT-path problem, not a filter-policy rejection.

Inbound port forwarding is DNAT. A client connects to the public server:

1

src=198.51.100.20:41000 dst=<public-ipv4>:443

The host rewrites the destination before routing:

1

src=198.51.100.20:41000 dst=10.187.54.3:443

I use my own small tool, iptfwd, to manage these rules from a config file: charlie0129/iptfwd.

Example config for this topology:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


defaults:
 public_iface: eno1
 private_iface: vmbr0
 manage_filter: true
 enable_ip_forwarding: true

nat:
 - name: intranet-v4
 source: 10.187.54.0/24
 type: masquerade

 - name: intranet-v6
 source: fc10:187:54::/64
 type: snat
 to_source: <public-ipv6>

rules:
 - name: service-http-v4
 proto: tcp
 public_port: 80
 target: 10.187.54.3
 target_port: 80

 - name: service-https-v4
 proto: tcp
 public_port: 443
 target: 10.187.54.3
 target_port: 443

 - name: service-http-v6
 proto: tcp
 public_port: 80
 target: fc10:187:54::3
 target_port: 80

 - name: service-https-v6
 proto: tcp
 public_port: 443
 target: fc10:187:54::3
 target_port: 443

For IPv4, MASQUERADE is convenient. For NAT66, I prefer explicit SNAT to the public IPv6 address. IPv6 interfaces can have multiple addresses, and explicit SNAT makes the translated source predictable.

If you do not want to use iptfwd, the equivalent basic iptables rules are:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


iptables -t nat -A POSTROUTING -s 10.187.54.0/24 -o eno1 -j MASQUERADE
iptables -A FORWARD -i vmbr0 -o eno1 -j ACCEPT
iptables -A FORWARD -i eno1 -o vmbr0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

public_ipv6=<YOUR_PUBLIC_IPV6>
ip6tables -t nat -A POSTROUTING -s fc10:187:54::/64 -o eno1 -j SNAT --to-source "$public_ipv6"
ip6tables -A FORWARD -i vmbr0 -o eno1 -j ACCEPT
ip6tables -A FORWARD -i eno1 -o vmbr0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

apt install iptables-persistent
netfilter-persistent save

Before setting up DHCP, test NAT manually:

Create a CT on vmbr0.
Set IPv4 to something like 10.187.54.100/24.
Set IPv4 gateway to 10.187.54.1.
Set IPv6 to something like fc10:187:54::100/64.
Set IPv6 gateway to fc10:187:54::1.
Test:

1
2


ping www.google.com
ping6 www.google.com

If that fails, debug NAT before adding DHCP.

DHCP and DNS with Pi-hole

I use Pi-hole as DHCP and DNS for the private bridge because it has a nice web UI and makes static leases easy.

Create a small Alpine CT:

Bridge: vmbr0.
IPv4: 10.187.54.2/24.
IPv4 gateway: 10.187.54.1.
IPv6: fc10:187:54::2/64.
IPv6 gateway: fc10:187:54::1.
CPU: 1 core is enough.
RAM: 128 MB is enough.
Disk: 512 MB is enough.

Install Pi-hole:

1
2
3
4


apk update
apk add curl bash
curl -sSL https://install.pi-hole.net | bash
# You may disable query logs to reduce writes to the database.

If you forget the web password:

1

pihole setpassword

Reduce query-log database writes:

1

pihole-FTL --config database.DBinterval 1800

In the Pi-hole UI:

Enable DHCP for 10.187.54.0/24.
Set the router/gateway to 10.187.54.1.
Configure DNS as desired.
Disable NTP sync under Settings -> All Settings -> Network Time Sync. Uncheck ntp.ipv4/ipv6/sync.active. An unprivileged CT cannot set system time anyway.

Alpine log cleanup I use for small CTs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


cat >/etc/periodic/hourly/logtruncate <<'EOF'
#!/bin/sh

LOG_DIR="/var/log"
MAX_SIZE=$((4 * 1024 * 1024))

find "$LOG_DIR" -type f ! -name '*.0' | while read -r file; do
 size=$(stat -c %s "$file" 2>/dev/null || stat -f %z "$file" 2>/dev/null || echo 0)

 if [ "$size" -gt "$MAX_SIZE" ]; then
 logger -t logrotate "Rotating $file ($size bytes)"
 cp "$file" "$file.0"
 truncate -s 0 "$file"
 logger -t logrotate "Rotated $file"
 fi
done
EOF

chmod +x /etc/periodic/hourly/logtruncate
sed -i 's/^SYSLOGD_OPTS=.*/SYSLOGD_OPTS="-t -s 0"/' /etc/conf.d/syslog

cat >/etc/periodic/daily/apkcacheclean <<'EOF'
#!/bin/sh
find /var/cache/apk -type f -name '*.apk' -mtime +7 -delete
find /var/cache/apk -type f -name '*.tar.gz' -mtime +90 -delete
EOF

chmod +x /etc/periodic/daily/apkcacheclean

Host Monitoring

I keep monitoring tools in Podman containers instead of installing everything directly on the Proxmox host.

Install Podman:

1
2
3
4
5
6


# So Intel PCM can work.
echo msr >> /etc/modules
modprobe msr

# Install podman to run monitoring tools in containers. Do not include any unnecessary packages to keep the host clean and minimal. Remember to install aardvark-dns to make container DNS work.
apt install --no-install-recommends podman aardvark-dns

On current Debian/Proxmox, Podman should use overlay storage on ZFS:

1

podman info --format '{{.Store.GraphDriverName}}'

Expected:

1

overlay

Make podman-restart.service also handle containers with restart-policy=unless-stopped:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


mkdir -p /etc/systemd/system/podman-restart.service.d

cat >/etc/systemd/system/podman-restart.service.d/override.conf <<'EOF'
[Service]
ExecStart=/usr/bin/podman $LOGGING start --all --filter restart-policy=always --filter restart-policy=unless-stopped
ExecStop=/usr/bin/podman $LOGGING stop --all --filter restart-policy=always --filter restart-policy=unless-stopped
EOF

systemctl daemon-reload
systemctl enable podman-restart.service

Enable the Podman socket and Docker-compatible CLI:

1
2
3
4
5


systemctl enable --now podman.socket

apt install --no-install-recommends docker-cli docker-compose
docker context create podman --docker "host=unix:///run/podman/podman.sock"
docker context use podman

An example monitoring compose file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85


services:
 vmagent:
 container_name: vmagent
 image: victoriametrics/vmagent:v1.144.0
 restart: unless-stopped
 mem_limit: 512M
 environment:
 GOMEMLIMIT: 900MiB
 command:
 - -httpListenAddr=0.0.0.0:8428
 - -cacheExpireDuration=5m
 - -promscrape.config=/etc/vmagent/scrape.yml
 - -remoteWrite.url=https://XXX # Your VictoriaMetrics remote write endpoint
 - -remoteWrite.basicAuth.username=XXX
 - -remoteWrite.basicAuth.password=XXX
 - -remoteWrite.maxDiskUsagePerURL=128MiB
 - -remoteWrite.tmpDataPath=/var/vmagent
 - -remoteWrite.vmProtoCompressLevel=3
 - -remoteWrite.flushInterval=66s
 - -remoteWrite.label=instance=XXX
 extra_hosts:
 - "host.docker.internal:host-gateway"
 volumes:
 - ./vmagent/config:/etc/vmagent
 - ./vmagent/data:/var/vmagent

 smartctlexporter:
 container_name: smartctlexporter
 image: prometheuscommunity/smartctl-exporter:v0.14.0
 restart: unless-stopped
 mem_limit: 64M
 user: 0:0
 environment:
 GOMEMLIMIT: 110MiB
 privileged: true

 nodeexporter:
 container_name: nodeexporter
 image: prom/node-exporter:v1.11.1
 restart: unless-stopped
 mem_limit: 64M
 environment:
 GOMEMLIMIT: 110MiB
 privileged: true
 command:
 - --path.rootfs=/host
 - --web.listen-address=:9100
 network_mode: host
 pid: host
 volumes:
 - "/:/host:ro,rslave"

 podmanexporter:
 container_name: podmanexporter
 image: quay.io/navidys/prometheus-podman-exporter:v1.21.0
 mem_limit: 64M
 user: 0:0
 privileged: true
 environment:
 GOMEMLIMIT: 110MiB
 CONTAINER_HOST: unix:///run/podman/podman.sock
 restart: unless-stopped
 volumes:
 - /run/podman/podman.sock:/run/podman/podman.sock
 command:
 - --web.listen-address=:9882

 cgroupexporter:
 container_name: cgroupexporter
 image: ghcr.io/arianvp/cgroup-exporter:0.3.3-amd64
 mem_limit: 64M
 environment:
 GOMEMLIMIT: 110MiB
 restart: unless-stopped
 volumes:
 - /sys/fs/cgroup:/sys/fs/cgroup:ro
 command:
 - -listen-address=:3232

 pcm:
 container_name: pcm
 image: opcm/pcm
 mem_limit: 96M
 restart: unless-stopped
 privileged: true

Service CT

I keep public-facing services in a separate CT and forward only selected ports from the host.

Create an unprivileged Alpine CT on vmbr0, then:

1
2
3


apk add openssh openssh-server
rc-update add sshd default
rc-service sshd start

Add this to /etc/ssh/sshd_config:

1
2
3


PermitRootLogin yes
ClientAliveInterval 60
ClientAliveCountMax 3

Install Podman inside the CT:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


cat >/usr/local/bin/enablecgroup2nesting <<'EOF'
#!/bin/sh
if [ -f /sys/fs/cgroup/cgroup.controllers ]; then
 echo "Enabling cgroup v2 nesting"
 mkdir -p /sys/fs/cgroup/init
 xargs -rn1 < /sys/fs/cgroup/cgroup.procs > /sys/fs/cgroup/init/cgroup.procs || :
 sed -e 's/ / +/g' -e 's/^/+/' < /sys/fs/cgroup/cgroup.controllers \
 > /sys/fs/cgroup/cgroup.subtree_control
fi
EOF
chmod +x /usr/local/bin/enablecgroup2nesting

cat >/etc/init.d/cgroup2nesting <<'EOF'
#!/sbin/openrc-run

description="Enable nesting of cgroup2."

depend()
{
 keyword -docker -podman -prefix -systemd-nspawn -vserver -wsl
 after sysfs
}

start()
{
 ebegin "Enabling cgroup v2 nesting"
 /usr/local/bin/enablecgroup2nesting
 eend $?
}
EOF
chmod +x /etc/init.d/cgroup2nesting
rc-update add cgroup2nesting default
rc-service cgroup2nesting start

apk add podman iptables
sed -i 's/^#log_size_max =.*/log_size_max = 1048576/' /etc/containers/containers.conf

rc-update add podman
rc-service podman start

apk add docker-cli docker-cli-compose
docker context create podman --docker "host=unix:///run/podman/podman.sock"
docker context use podman

Also check out the Pi-Hole setup section for Alpine log cleanup and other small CT optimizations. You must do this because Alpine does not have log rotation by default. You disk will fill up if you do not manage logs.

Then run Caddy or Traefik in that CT for reverse proxy and TLS termination. Forward 80 and 443 from the host to the service CT with iptfwd or iptables.

If you use a privileged CT with Podman over ZFS, you may need an overlay mount helper:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


cat >/usr/local/bin/zfsoverlaymount <<'EOF'
#!/bin/sh
/bin/mount -t overlay overlay "$@"
EOF
chmod +x /usr/local/bin/zfsoverlaymount

cat >/etc/containers/storage.conf <<'EOF'
[storage.options]
mount_program = "/usr/local/bin/zfsoverlaymount"
EOF

I prefer unprivileged CTs unless there is a specific reason not to use them.

SMTP Notifications

By default, Proxmox may send mail directly and anonymously to the email address you entered during installation. Some providers reject that mail.

Configure a real SMTP target instead:

1

Datacenter -> Notifications -> Notification Targets -> Add -> SMTP

Final State

The host ended up with:

Proxmox VE 9.2.
Mirrored ZFS root pool on 128 GiB from each NVMe SSD.
Striped ZFS scratch pool on the remaining SSD space.
4K LBA on both NVMe drives.
Private-only vmbr0 for VMs and CTs.
IPv4 NAT and IPv6 NAT66 from vmbr0 to eno1.
80/443 forwarded to a service CT.
Pi-hole handling private DHCP and DNS.
Podman-based host monitoring.

For 19.90 USD/month, this is a surprisingly useful little virtualization box.

Proxmox VE Shared GPU Installation Guide

Sat, 16 Aug 2025 20:49:00 +0800

Introduction

Most schools and universities in China have a limited number of GPU servers (often only one or two, or even none). This makes the management of GPU development machines quite different than corporation which have virtually unlimited GPU servers.

Let’s consider a normal lab environment with 12 students, each needing a GPU server for their projects, and we have one GPU server with 8 GPUs available.

If we give each student a dedicated GPU server, we would need 12 servers, this is impractical and costly.
If we let all the students share a single GPU server (all students have access to all resources), it would be very difficult to manage, as each student would need to install their own software and dependencies, leading to conflicts. Soon the GPU server will become a hot mess. Making each student use a different user account would not solve the problem, as they would still share the same OS and software environment, and will potentially break each other’s environment.
If we make use of IOMMU and PCIe passthrough, we can assign a virtual machine with one dedicated passthrough’d GPU to each student. This approach has almost no interference between students, as each student has their own OS and software environment, even with it’s own GPU. However, this leads to a lot of wasted resources, because most of the time the GPU is idle, and each student will only have access to one GPU at a time, even if all of the GPUs are idle.
If we use GPU virtualization, we can assign a virtual machine with a shared GPU to each student. This way, each student can use the GPU resources as needed, and the GPU can be shared among multiple students. This approach is more efficient and cost-effective, as it allows for better resource utilization. However, it requires a GPU that supports virtualization, such as NVIDIA’s vGPU or AMD’s MxGPU. Most consumer GPUs do not support virtualization, so this approach is not feasible for most schools and universities.

This is where containerization comes in. By using containerization, we can create a shared GPU development server that runs a container for each student. Each container can have its own software environment, and all (or some of) of the GPUs can be shared among multiple containers. This approach is more efficient and cost-effective, as it allows for better resource utilization, and does not require a GPU that supports virtualization. However, it should be noted that this approach is not as isolated as virtual machines, as all containers share the same kernel and GPU resources. Therefore, it is important to ensure that the containers are properly configured, and that the students are aware of the limitations and potential issues that may arise from sharing the same GPU resources (for example, if one student runs a GPU-intensive task, it may affect the performance of other students’ containers).

We will be using the CT (LXC Containers) in Proxmox VE to achieve this. Why not use Docker? Because Docker is meant for running applications, not for running full Linux distros. Although you can use something like sysbox to run a full Linux distro in Docker, i would still prefer to use LXC Containers, as they are built for this purpose and are already built into Proxmox VE, making it easier to manage and deploy.

Proxmox VE Installation

Download the latest Proxmox VE ISO from the official website. At the time of writing, the latest version is Proxmox VE 9.0. Use whatever method you prefer to install Proxmox VE, such as using a USB drive.

I will use IPMI of the server to install Proxmox VE, as it is the most convenient method for me. You can also use a monitor and keyboard to install Proxmox VE if you prefer.

Mount the Proxmox VE installation ISO as virtual media so we can boot into.

Now reboot the server into the virtual media. Supermicro motherboards lets you invode the boot menu using F11. Your motherboard may have a different shortcut.

Choose our CDROM virtual media.

Boot into the Proxmox install menu (I prefer terminal UI over GUI ones).

After you accepted the license, you will need to choose the target installtion disk. You should choose the boot SSDs on your server, not data drives. The drive in picture is 3*1.92TB SSDs in RAID 5.

Toggle Advanced options. Keep ext4, we will not be using ZFS. I know ZFS has CoW, snapshotting, compressing, checksums, and a ton of other features. But we will not use ZFS, because it will potentially cause some problems, e.g., high IO load when using RAID-Z zvols, extremely slow container start times when using ZFS storage driver in Docker, SSD write amplifications, low SSD random read/write performance, ZFS ARC not being given back to the OS as fast as needed on high memory pressure systems leading to OOM, and a ton of other issues that I previously encountered. To save me some trouble, I will use the battle-tested ext4.

Total size: keep as-is
Swap size: 0, we don’t need it, our memory is large enought (1TiB) and we will later use ZRAM as swap
Maximum root volume size: 100 (GiB), make it slightly larger so we can install things into the root volume, but not too large to occupy the data volume space.

Other options can be left empty.

Other install steps (Keyboard setup, root password) can proceed as you normally would do. Remember to set a complex root password.

Regarding to IP addresses:

If you are going to use a static IP address like I do, you can just set it here and forget about it.

If you are going to use a dynamic (DHCP) addresses, you should keep what’s already in here (static IP). You don’t need to change anything. The valid IP address should already be automatically discovered from DHCP servers and filled in. After the installation, you can change the static IP mode to DHCP mode by running vi /etc/network/interfaces and make the following edits:

1
2
3
4
5
6
7
8


 auto vmbr0
- iface vmbr0 inet static
+ iface vmbr0 inet dhcp # Enable DHCP and remove static IP.
- address 10.112.154.220/16
- gateway 10.112.0.1
 bridge-ports eno1
 bridge-stp off
 bridge-fd 0

After you enabled DHCP in the config, apply it by ifreload -a. Check if it takes effect by ip a show vmbr0, you should see something like valid_lft 7160sec preferred_lft 7160sec. If there is no xx sec inside it, your DHCP is malfunctioning.

1
2
3
4
5
6
7


root@z8:/etc/network# ip a show vmbr0
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
 link/ether 30:13:8b:6d:3a:a6 brd ff:ff:ff:ff:ff:ff
 inet 10.112.154.220/16 brd 10.112.255.255 scope global dynamic vmbr0
 valid_lft 7160sec preferred_lft 7160sec
 inet6 fe80::3213:8bff:fe6d:3aa6/64 scope link
 valid_lft forever preferred_lft forever

Proxmox VE Post-Installtion

Change APT Sources

In China, we don’t really have a good global Internet connection, so we will change APT source to mirrors in China.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# Change APT source mirrors
sed -i 's/deb.debian.org/mirrors.ustc.edu.cn/g' /etc/apt/sources.list.d/debian.sources
sed -i 's/security.debian.org/mirrors.ustc.edu.cn/g' /etc/apt/sources.list.d/debian.sources

# Change Ceph source mirrors
if [ -f /etc/apt/sources.list.d/ceph.sources ]; then
 CEPH_CODENAME=`ceph -v | grep ceph | awk '{print $(NF-1)}'`
 source /etc/os-release
 cat > /etc/apt/sources.list.d/ceph.sources <<EOF
Types: deb
URIs: https://mirrors.ustc.edu.cn/proxmox/debian/ceph-$CEPH_CODENAME
Suites: $VERSION_CODENAME
Components: no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF
fi

Remove enterprise repo and add no-subscription repo.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Remove enterprise sources
rm /etc/apt/sources.list.d/pve-enterprise.sources

# Add no-subscription sources
cat > /etc/apt/sources.list.d/pve-no-subscription.sources <<EOF
Types: deb
URIs: https://mirrors.ustc.edu.cn/proxmox/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF

Change CT Template Sources

Use mirrors in China.

1
2


sed -i.bak 's|http://download.proxmox.com|https://mirrors.ustc.edu.cn/proxmox|g' /usr/share/perl5/PVE/APLInfo.pm
systemctl restart pvedaemon

Stop Cluster Services

We are not using PVE clusters. Disable them. If you are using PVE clusters, you probably will not be reading this guide :P

1
2
3


systemctl disable --now pve-ha-crm.service
systemctl disable --now pve-ha-lrm.service
systemctl disable --now corosync.service

Install Common Tools

Useful tools that will be used regularly.

1

apt install htop sysstat vim sudo

Configure Shell

I like ZSH and my dotfiles, so I will use them.

1

apt install git zsh

1
2
3
4
5
6
7
8


cd
git clone --depth=1 https://github.com/charlie0129/dotfiles.git
cd dotfiles
./bootstrap.sh -f
chsh -s /usr/bin/zsh

# Run zsh and follow the instructions
zsh

Configure ZRAM Swap

1
2
3
4
5


git clone --depth=1 https://github.com/foundObjects/zram-swap.git
cd zram-swap
./install.sh
cd ..
rm -rf zram-swap

The default lz4 is a good balance between speed and compression ratio. If you want to sacrifice speed for better compression, you can change the config to use zstd as the compression method.

1

sed -i 's/_zram_algorithm=.*/_zram_algorithm="zstd"/g' /etc/default/zram-swap

Apply

1

systemctl restart zram-swap

Tune kernel parameters to make better use of ZRAM

1
2
3
4
5
6


cat <<EOF > /etc/sysctl.d/zram.conf
vm.swappiness = 180
vm.watermark_boost_factor = 0
vm.watermark_scale_factor = 125
vm.page-cluster = 0
EOF

Apply

1

sysctl --system

Enable IPv6 SLAAC

Proxmox disables IPv6 by default. To enable IPv6 SLAAC:

1
2
3
4
5
6
7
8


cat <<EOF > /etc/sysctl.d/ipv6.conf
net.ipv6.conf.default.accept_ra = 2
net.ipv6.conf.all.accept_ra = 2
net.ipv6.conf.default.forwarding = 1
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.proxy_ndp = 1
net.ipv6.conf.all.proxy_ndp = 1
EOF

Apply it

1
2
3


sysctl --system
# Note: the following command may break your internet connection.
systemctl restart networking

TRIM Optimizations

TRIM will help prolong SSD lifespan and maintain performance. Enable it on the LVM thin pool.

1

sed -i 's/.*issue_discards = .*/\tissue_discards = 1/g' /etc/lvm/lvm.conf

Since LXCs are just on the host, it will automatically issue discards if the underlying storage supports it, so no further action is needed. If you are using VM, be sure to use SCSI disks (VirtIO SCSI Single) with discard=on option. I’ve heard that VirtIO Block devices will not work with dicards but I haven’t tested it yet. Correct me if I’m wrong.

Limit Journal Size

1

sed -i 's/.*SystemMaxUse.*/SystemMaxUse=32M/g' /etc/systemd/journald.conf

Apply

1
2


systemctl daemon-reload
systemctl restart systemd-journald

NVIDIA P2P Driver Installation

Instead of regular NVIDIA drivers, we will be installing P2P-enabled drivers to force enable PCIe P2P capabilities on consumer cards (like GeForce RTX 4090). This will bring performance boost (~10%) across multiple scenarios. For details, refer to my blog post: Enabling PCIe P2P on NVIDIA RTX 4090s

Disable IOMMU

Since PCIe P2P in Linux doesn’t work so well with IOMMU-enabled systems (there are many potential issues you may run into), you may as well just disable IOMMU. Note that if you have more than 255 CPU cores, you will only have access to 255 CPUs due to APIC fallback.

To do disable IOMMU, you can either

Disable Intel VT-d (AMD is on by default) in BIOS
Disable intel_iommu or amd_iommu in Linux kernel parameters

To disable intel_iommu or amd_iommu in Linux kernel parameters, use the following command to add intel_iommu=off and amd_iommu=off to your GRUB_CMDLINE_LINUX_DEFAULT. BTW: I also reduce the screen resolution to 1024x768 because it’s a server

1

sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=off amd_iommu=off video=1024x768@60"/g' /etc/default/grub

Reboot your system to see the effect.

1

reboot

Make sure the follow command produces NO entries. If there are output, it means IOMMU is not correctly disabled.

1

ls /sys/class/iommu

Disable ACS

You may need to disable ACS to get PCIe P2P to work, refer to https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html (PCI Access Control Services)

Enable Above 4G Decoding

PCIe P2P will need to access each GPU’s memory, so the PCIe BAR should be large enough to cover GPU’s memory. You must enable Above 4G Decoding in your motherboard settings to make the PCIe BAR large enough to work. Enable it in your motherboard’s BIOS.

Install Official Driver

Before we install the P2P kernel module, we need to install the official drivers.

Since we are installing unofficial P2P kernel modules, there is a limited number of driver version that will work. The version of the P2P kernel modules MUST be the same as the official driver we are going to install.

Refer to https://github.com/tinygrad/open-gpu-kernel-modules to see what versions are available. I also have my patches that works with much newer driver versions (575.57.08) (at the time of writing) https://github.com/charlie0129/open-gpu-kernel-modules .

I will be using version 575.57.08, so we should download the official driver of the same version (575.57.08). You should download the .run file, e.g., NVIDIA-Linux-x86_64-575.57.08.run

1
2
3


# Assume you download the installer as NVIDIA-Linux-x86_64-575.57.08.run
installer="NVIDIA-Linux-x86_64-575.57.08.run"
chmod +x $installer

Skip kernel modules because we will install P2P-patched version later.

1

./$installer --no-kernel-modules

Choose whatever you want, doesn’t matter.

Yes, disable nouveau, we will be using NVIDIA drivers.

Do not abort, just continue. Nouveau will be disabled on next boot.

Just choose the default option for every step that comes later.

Remove the driver installation file because we will not be using it later.

1

rm $installer

You can skip reboot now. We will reboot after we installed the kernel modules.

Install P2P-Enabled Kernel Modules

Make sure the version of the P2P-enabled kernel modules match the version of the driver. I will use veriosn 575.57.08.

1

git clone --depth=1 -b 575.57.08-p2p https://github.com/charlie0129/open-gpu-kernel-modules.git

To build kernel modules, install build dependencies and kernel source. sudo installed because the install script uses sudo but we don’t have it now.

1

apt install sudo build-essential proxmox-headers-$(uname -r)

Build and install the kernel module

1

./install.sh

You can safely ignore the NVIDIA-SMI failure as long as the build succeeds (you should see a DEPMOD /lib/modules/6.14.8-2-pve line) because we haven’t rebooted yet (so NVIDIA-SMI can’t be used).

Remove the source after installing

1
2


cd ..
rm -rf open-gpu-kernel-modules

Reboot your system and you should see nvidia-smi running fine.

To tell if P2P is enabled, we will use a simple method (performance testing will be done later):

1

nvidia-smi -q | grep -i bar -A 3

You should see > 2048 MiB BAR1 Total memory, 32768 MiB in my case.

1
2
3
4


 BAR1 Memory Usage
 Total : 32768 MiB
 Used : 2 MiB
 Free : 32766 MiB

Enable Persistence Mode

Persistence mode will:

Keep the GPU driver running so program can start faster
Lower GPU power mode when it’s idle to save power. For RTX 4090s, it can drop from ~70W to ~10W.
Make sure /dev/nvidia* device nodes are ready. This is useful because we are passing the GPU devices /dev/nvidia* to CTs (LXC Containers) later, it requires the device nodes to be present on start up for auto-started CTs.

So you really should enable it.

To enable persistence mode:

1

/usr/bin/nvidia-smi -pm 1

To make it persistent across reboots, we will use a cron job.

1

crontab -e

Add a line to run nvidia-smi on boot:

1

@reboot /usr/bin/nvidia-smi -pm 1 >/dev/null 2>&1

Why not use nvidia-persistenced systemd service? Because for whatever reason, it does not initialize the device nodes /dev/nvidia* correctly, so auto-started CTs will encounter Cuda failure 'unknown error'.
1
2
3
4
cd /usr/share/doc/NVIDIA_GLX-1.0/samples
tar jxf nvidia-persistenced-init.tar.bz2
cd nvidia-persistenced-init
./install.sh

You should feel nvidia-smi runs much faster than before.

Build CT Templates

We will build a CT Template that has everything a student will need:

Common build dependencies
CUDA
Docker
…

Download the Linux distro you want. I chose Ubuntu 24.04, not because I like Ubuntu (I use Debian), but because most students only know Ubuntu.

Create an unprivileged CT just like you normally would. Just remember to give it a bit more disk space (32GB or more) because we will install CUDA later and CUDA is really large.

Install Docker

After you created the CT, shut it down. Add additional settings to allow Docker to use overlayfs driver, otherwise Docker images will be extremely inefficient in CTs.

1
2
3
4
5
6
7


# This command should be run on the host, not CT.
# Assume 8000 is the ID of your template CT
cat <<EOF >> /etc/pve/local/lxc/8000.conf
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
EOF

All following commands should be run in the CT unless specified otherwise.

Add Docker configuration. Systemd journal is used as log driver to make use of previously configured journal size limit. By writing logs to a centralized location, it’s also useful if you want to write logs to memory to reduce disk writes (e.g. log2ram).

1
2
3
4
5
6
7
8


mkdir -p /etc/docker/
cat <<EOF > /etc/docker/daemon.json
{
 "live-restore": true,
 "experimental": true,
 "log-driver": "journald"
}
EOF

Install Docker

1
2


export DOWNLOAD_URL=https://mirrors.ustc.edu.cn/docker-ce
curl -fsSL https://get.docker.io | sh

Revert Containerd Config

Docker modified containerd config. We will revert it to the default config.

1

containerd config default >/etc/containerd/config.toml

Add GPUs to CT

Choose which GPU you need to passthrough by looking for the index in nvidia-smi, or by ls -l /dev/nvidia*

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# This command should be run on the host, not CT.
# ls -l /dev/nvidia*
crw-rw-rw- 1 root root 195, 0 2025-08-17 01:47:58 /dev/nvidia0
crw-rw-rw- 1 root root 195, 1 2025-08-17 01:48:00 /dev/nvidia1
crw-rw-rw- 1 root root 195, 2 2025-08-17 01:48:01 /dev/nvidia2
crw-rw-rw- 1 root root 195, 3 2025-08-17 01:48:02 /dev/nvidia3
crw-rw-rw- 1 root root 195, 4 2025-08-17 01:48:03 /dev/nvidia4
crw-rw-rw- 1 root root 195, 5 2025-08-17 01:48:05 /dev/nvidia5
crw-rw-rw- 1 root root 195, 6 2025-08-17 01:48:06 /dev/nvidia6
crw-rw-rw- 1 root root 195, 7 2025-08-17 01:48:07 /dev/nvidia7
... (omitted)

For example, if want to add all 8 GPUs (0, 1, 2, 3, 4, 5, 6, 7) to CT, I will need to run:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# This command should be run on the host, not CT.
# Assume 8000 is the ID of your template CT
cat <<EOF >> /etc/pve/local/lxc/8000.conf
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia1 dev/nvidia1 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia2 dev/nvidia2 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia3 dev/nvidia3 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia4 dev/nvidia4 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia5 dev/nvidia5 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia6 dev/nvidia6 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia7 dev/nvidia7 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/fb0 dev/fb0 none bind,optional,create=file
EOF

Note the lxc.mount.entry: /dev/nvidiaX dev/nvidiaX none bind,optional,create=file lines. Each line represents a single GPU to add to the CT. I will add 8 GPUs, so I add 8 lines from nvidia0 through nvidia7. Remove or add them as needed.

Check if the GPUs are there inside CT:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# ls -l /dev/nvidia*
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:00 /dev/nvidia-modeset
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:50:43 /dev/nvidia-uvm
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:50:43 /dev/nvidia-uvm-tools
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:47:58 /dev/nvidia0
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:00 /dev/nvidia1
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:01 /dev/nvidia2
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:02 /dev/nvidia3
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:03 /dev/nvidia4
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:05 /dev/nvidia5
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:06 /dev/nvidia6
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:07 /dev/nvidia7
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:47:58 /dev/nvidiactl

We haven’t installed drivers yet, so nvidia-smi is not available.

Install GPU Driver

Important: you should use the same driver version as the host (575.57.08 in my case).

1
2
3
4


wget <driver-runfile-url> -O driver.run
chmod +x driver.run
driver.run --no-kernel-modules
rm driver.run

Just install as normal.

Now you should be able to use nvidia-smi and see the GPUs you added to the CT (8 GPUs in my case).

Install CUDA

Important: current driver version MUST be greater than (or equal to) the driver required by CUDA. For example, CUDA 12.9.0 will work on my machine because CUDA 12.9.0 requires 575.00, and my current driver version is 575.57.08 (greater than 575.00).

1
2
3
4


wget <cuda-runfile-url> -O cuda.run
chmod +x cuda.run
cuda.run
rm cuda.run

Remember to deselect the driver (we already installed the correct driver):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


┌──────────────────────────────────────────────────────────────────────────────┐
│ CUDA Installer │
│ - [ ] Driver <== deselect this │
│ [ ] 575.51.03 <== deselect this │
│ + [X] CUDA Toolkit 12.9 │
│ [ ] CUDA Demo Suite 12.9 <== deselect this (not useful, takes up space) │
│ [ ] CUDA Documentation 12.9 <== deselect this (not useful, takes up space) │
│ - [ ] Kernel Objects │
│ [ ] nvidia-fs │
│ Options │
│ Install │
│ │
│ Up/Down: Move | Left/Right: Expand | 'Enter': Select | 'A': Advanced options │
└──────────────────────────────────────────────────────────────────────────────┘

After CUDA installation, configure PATH and LD_LIBRARY_PATH per printed instructions.

1
2
3
4
5


... (omitted)
Please make sure that
 - PATH includes /usr/local/cuda-12.9/bin
 - LD_LIBRARY_PATH includes /usr/local/cuda-12.9/lib64, or, add /usr/local/cuda-12.9/lib64 to /etc/ld.so.conf and run ldconfig as root
... (omitted)

Note that you can use /usr/local/cuda instead of /usr/local/cuda-XX.X so it’s independent of CUDA versions.

Also add export CUDA_HOME=/usr/local/cuda to your shell rc file.

nvcc should be available after a shell reload:

1
2
3
4
5
6


# nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Apr__9_19:24:57_PDT_2025
Cuda compilation tools, release 12.9, V12.9.41
Build cuda_12.9.r12.9/compiler.35813241_0

Install NCCL

You will need NCCL for most GPU-related communications.

You should find a NCCL that’s compatible with your CUDA version. For example, NCCL 2.27.3 is compatible with my CUDA version 12.9.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


# Download NCCL package
wget https://xxx/nccl_2.27.3-1+cuda12.9_x86_64.txz

tar Jxf nccl_2.27.3-1+cuda12.9_x86_64.txz

# Set some variables
NCCL_VERSION=2.27

mkdir -p /usr/local/nccl-$NCCL_VERSION

cp -vRf nccl_2.27.3-1+cuda12.9_x86_64/* /usr/local/nccl-$NCCL_VERSION
rm -rf nccl_2.27.3-1+cuda12.9_x86_64

ln -s /usr/local/nccl-$NCCL_VERSION /usr/local/nccl

echo "/usr/local/nccl/lib" >>/etc/ld.so.conf
ldconfig

ln -s /usr/local/nccl/include/nccl.h /usr/local/include/nccl.h

# Remove NCCL package
rm https://xxx/nccl_2.27.3-1+cuda12.9_x86_64.txz

Also add export NCCL_HOME=/usr/local/nccl in your shell rc file,

Install NVIDIA Container Toolkit

To run Docker containers with GPUs.

Note that the URLs below are mirrors in China. You can use original URL if you prefer.

1
2
3
4
5
6
7
8


curl -fsSL https://mirrors.ustc.edu.cn/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://mirrors.ustc.edu.cn/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
 sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
 tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sed -i 's/nvidia.github.io/mirrors.ustc.edu.cn/g' /etc/apt/sources.list.d/nvidia-container-toolkit.list

apt-get update
apt-get install -y nvidia-container-toolkit

Configure Docker and Containerd to use NVIDIA Container Toolkit

1
2


nvidia-ctk runtime configure --runtime=docker
nvidia-ctk runtime configure --runtime=containerd

Since we are running inside unprivileged containers, we need to set no-cgroups to true

1

nvidia-ctk config --set nvidia-container-cli.no-cgroups --in-place

Install NVIDIA Nsight Systems

So we can profile applications running on the GPU.

1
2
3


wget https://xxx/NsightSystems-linux-cli-public-2025.3.1.90-3582212.deb
dpkg -i NsightSystems-linux-cli-public-2025.3.1.90-3582212.deb
rm -f NsightSystems-linux-cli-public-2025.3.1.90-3582212.deb

Functionality Tests

NCCL

1
2
3
4


git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
git reset --hard 903918f # I only tested this commit.
make

P2P ON: NCCL_P2P_LEVEL=sys ./build/all_reduce_perf --minbytes 8 --maxbytes 128M --stepfactor 2 --ngpus 8 (change --ngpus accordingly)

P2P OFF: NCCL_P2P_DISABLE=1 ./build/all_reduce_perf --minbytes 8 --maxbytes 128M --stepfactor 2 --ngpus 8 (change --ngpus accordingly)

You should see a bandwidth increase after P2P is on.

CUDA P2P

1
2
3
4
5


git clone https://github.com/NVIDIA/cuda-samples.git
git reset --hard 9c688d7 # I only tested this commit.
cd cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest
make
./p2pBandwidthLatencyTest

You should see lower GPU-GPU latency, higher bandwidth if P2P is on.

Prepare for Templating

Check if there are unnecessary files and remove them.

1
2


cd
ls -l

Remove history

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


cd

unset HISTFILE
rm .*_history

echo -n >/var/log/lastlog
echo -n >/var/log/wtmp
echo -n >/var/log/btmp

journalctl --vacuum-time=1s

rm -f /var/log/*.log

Now, it’s ready for converting to a template. To give each student a new development container, just clone this template.

Other Minor Settings

RAID Card Write Back

To achieve maximum write speed, you can enable write back mode on your RAID card, so data is written to cache first. This will significantly (yes, by A LOT, sometimes over 100x) improve random write speed on HDDs (SSDs are already very fast). If your RAID card has a battery, you don’t need to worry about data integrity in case a power failure.

Write cache policy:

Write Throuh: no cache
Write Back: cached if battery backup is present on the RAID card
Always Write Back: always cached

Obstacles

Only 255 Cores are Recognized - x2APIC

I noticed that I have one CPU offline. It should recognize 256 CPUs but I only have 255 CPUs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# lscpu
Architecture: x86_64
 CPU op-mode(s): 32-bit, 64-bit
 Address sizes: 48 bits physical, 48 bits virtual
 Byte Order: Little Endian
CPU(s): 256
 On-line CPU(s) list: 0-254
 Off-line CPU(s) list: 255 <--------------- one CPU is offline -----------
Vendor ID: AuthenticAMD
 Model name: AMD EPYC 7763 64-Core Processor
 CPU family: 25
 Model: 1
 Thread(s) per core: 2
 Core(s) per socket: 64
 Socket(s): 2

After some digging, I learned that I need to enable x2APIC on both the motherboard and kernel to have more than 255 CPU cores recognized.

In BIOS Local APIC Mode should already be x2APIC .

Now I realized it can be because I disabled IOMMU for PCIe P2P before. It turns out that disableing IOMMU can limit the number of available logical cores to 255. The reason is that the Linux kernel disables x2APIC in this case and falls back to APIC, which can only enumerate a maximum of 255 (logical) cores.

I will keep IOMMU disabled because I need PCIe P2P to function (I don’t want to deal with P2P with IOMMU on). Just forget about the 256th core :p.

Share NVIDIA GPU between CTs in Proxmox VE

Mon, 04 Nov 2024 12:57:00 +0800

Background

Consider a small lab, students need to use GPU for their projects. We have a NVIDIA GPU in our Proxmox VE server, and we want to share the GPU between multiple containers so that multiple students can use the GPU at the same time.

Why not use a VM? Because a GPU can only be passed through to one VM at a time (only one student can use the GPU at a time). And resources are not flexible in VMs.

Why not create multiple users in the host and let them run their programs in the host? Because we want to isolate the students from the host, so that they can’t access the host and other students’ data.

Why not use Docker? Because Docker containers doesn’t have a full init system, and it’s hard to run some applications.

Since we use PVE and it has LXC containers built-in (called CT), it is a perfect choice.

Install Drivers on the Host

Make sure the GPU is detected by the host. Note the NVIDIA GPUs 3b:00.0 (Your address may differ).

1
2
3
4
5


# lspci | grep -i nvidia
3b:00.0 VGA compatible controller: NVIDIA Corporation TU104GL [Quadro RTX 5000] (rev a1)
3b:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)
3b:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1)
3b:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)

You may ask: does your entire lab only own one RTX 5000? What kind of lab is this? Are you cave people?

Yes, although we have multiple projects worth over millions of Chinese Yuan, most of the money is gone to the some other places (which I cannot publicly speak on the Internet 🤫 ). And the professors have no emphasis on students’ growth. As a result, we are actually poor as hell.

Since almost no one knows how to properly configure a Linux server, I want to help my classmates to learn more and let them use the only GPU. But to be honest, I won’t benefit from doing this. It’s just voluntary work.

Install prerequisites. Note that I am using pve-headers-$(uname -r) to install the headers for the current kernel. If you are using a different kernel, you may need to install the headers for that kernel. Also, you may want to use linux-headers-$(uname -r) instead of pve-headers-$(uname -r) if you are not using Proxmox VE.

1

# apt install -y gcc make pve-headers-$(uname -r)

Download CUDA toolkit from here and install it. Drivers are included in the CUDA toolkit so you don’t need to install drivers separately.

1
2


# wget <cuda-runfile-download-url>
# ./cuda_12.2.2_535.104.05_linux.run --silent

The default installation options will work fine. If anything fails, you can check the log file at /var/log/cuda-installer.log for CUDA logs and /var/log/nvidia-installer.log for NVIDIA driver logs.

PS: You need to blacklist nouveau driver. This is automatically done by PVE. If not, you can do this by creating a file /etc/modprobe.d/blacklist-nouveau.conf with the following content: blacklist nouveau. Then run update-initramfs -u to update the initramfs.

PPS: If you used to passthrough this GPU to a VM, be sure to remove the GPU from the VM’s hardware configuration in PVE otherwise PVE will bound the GPU to vfio-pci (see Kernel driver in use row in lspci -k) and cannot be used by the host.

PPPS: Some kernel versions are known to have problems with NVIDIA drivers. If you encounter problems, you may need to downgrade/upgrade the kernel. For example, kernel version 5.10.0 is known to have make[3]: *** No rule to make target 'scripts/module.lds', needed by '/tmp/selfgz38416/NVIDIA-Linux-x86_64-560.35.03/kernel-open/nvidia.ko' error.

After installation finished, check if the driver is loaded.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# nvidia-smi
Tue Nov 5 09:56:44 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Quadro RTX 5000 Off | 00000000:3B:00.0 Off | Off |
| 33% 44C P0 28W / 230W | 0MiB / 16384MiB | 6% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+

Allow NVIDIA Device Passthrough in CT

Now we need to allow the CT to access the GPU. I am using an unprivileged container here. Edit the CT’s configuration file (/etc/pve/local/lxc/<id>.conf). Add the following lines to the end of the file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


 arch: amd64
 cores: 4
 features: nesting=1
 hostname: ct-gpu-tmpl-deb127-cu122
 memory: 4096
 net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=AA:AB:F0:07:42:D0,ip=dhcp,type=veth
 ostype: debian
 rootfs: local-zfs:basevol-8001-disk-0,size=16G
 swap: 0
 unprivileged: 1
# These lines allow the container to access specific character devices (c) with rwm
# permissions (read, write, modify). These are needed for NVIDIA GPU access.
+ lxc.cgroup.devices.allow: c 195:* rwm
+ lxc.cgroup.devices.allow: c 509:* rwm
+ lxc.cgroup.devices.allow: c 235:* rwm
# These lines mount various GPU-related devices from the host into the container.
+ lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
+ lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
+ lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
+ lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
+ lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
+ lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
+ lxc.mount.entry: /dev/fb0 dev/fb0 none bind,optional,create=file

PS: If you cannot use nvidia-smi (it says Failed to initialize NVML: Unknown Error), there is a possibility that you are using cgroup2. Change all the lxc.cgroup.devices.allow lines to lxc.cgroup2.devices.allow.

Explanation:

Allows container access to NVIDIA device nodes:

c 195:* - NVIDIA character devices
c 509:* - NVIDIA UVM devices
c 235:* - NVIDIA CTL devices

Maps the following host GPU devices into container:

/dev/nvidia0 - Main GPU device
/dev/nvidiactl - NVIDIA control device
/dev/nvidia-modeset - Display mode setting
/dev/nvidia-uvm - Unified memory management
/dev/nvidia-uvm-tools - UVM diagnostic tools
/dev/dri - Direct Rendering Infrastructure
/dev/fb0 - Framebuffer device

Mount options:

bind: Mount as a bind mount
optional: Don’t fail if device doesn’t exist
create=file/dir: Create the mount point if it doesn’t exist

Note that if you are using a different GPU, you may need to change the device numbers. For example, /dev/nvidia1 instead of /dev/nvidia0. You can find the device numbers in nvidia-smi output.

Install Drivers in CT

Log into the CT. All the following commands are run in the CT.

You should be able to see NVIDIA devices inside the CT:

1
2
3
4
5
6


# ls -l /dev/nvidia*
---------- 1 root root 0 Nov 5 02:31 /dev/nvidia-modeset
crw-rw-rw- 1 nobody nogroup 507, 0 Nov 5 01:56 /dev/nvidia-uvm
crw-rw-rw- 1 nobody nogroup 507, 1 Nov 5 01:56 /dev/nvidia-uvm-tools
crw-rw-rw- 1 nobody nogroup 195, 0 Nov 5 01:56 /dev/nvidia0
crw-rw-rw- 1 nobody nogroup 195, 255 Nov 5 01:56 /dev/nvidiactl

Install CUDA and drivers, just like you would on a physical machine, except that you don’t need to install the kernel modules. I will install CUDA 12.2 (drivers are included in the CUDA installer).

1
2
3


# wget <cuda-runfile-download-url>
# apt install -y gcc
# ./cuda_12.2.2_535.104.05_linux.run --extract=$(pwd)/cu122

Note that I extracted the installer to manually install it because we want to skip kernel module installation and such options are not exposed in the installer.

Install the bundled drivers:

1
2


cd cu122
./NVIDIA-Linux-x86_64-535.104.05.run --no-nouveau-check --no-kernel-modules --silent

Run nvidia-smi to check if the driver is loaded.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# nvidia-smi
Tue Nov 5 05:49:02 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Quadro RTX 5000 Off | 00000000:3B:00.0 Off | Off |
| 33% 38C P0 23W / 230W | 0MiB / 16384MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+

We can now see that the GPU is accessible in the CT.

Let’s continue with the CUDA installation. Remember to uncheck the driver installation option because we have already installed the drivers above.

1

./cuda-linux.12.2.2-535.104.05.run

After a successful installation, you should add cuda binaries to PATH. Instructions should be printed at the end of the installation. Then you can run nvcc to see if CUDA is installed correctly.

Everything should be working by this point.

Missing `nvidia-uvm` and High Idle Power Draw

One problem I encountered is that when the host reboots, the GPU is not accessible in the CT. This is because nvidia-uvm device isn’t created until an application attempts to interact with the graphics card. This is a problem because no application will interact with the GPU at boot, so no nvidia-uvm device is created. But the CT needs the nvidia-uvm device bind-mounted at CT-startup in order to access the GPU.

Also, the graphics card have insanely high power draw at idle (over 100 Watts). The GPU is in P0 and never leaves it. We can use nvidia-persistenced to let the GPU enter a low-power state (P8) when not in use.

To solve this, we can run nvidia-smi -pm 1 (which enables nvidia-persistenced and keeps nvidia character device and handles frequency scaling) at boot. Add the following line to the host’s crontab to run nvidia-smi -pm 1 at boot.

PS: This only works if the host is a headless server (no monitor attached). If you have a monitor attached, you may need to run nvidia-smi below instead.

1
2


# crontab -e
@reboot /usr/bin/nvidia-smi -pm 1

Downsides

Despite the fact that this method works best for us, there are some downsides:

The CT will have full access to the GPU. If one CT uses all the GPU memory, other CTs will be starving. So you must trust the users of the CTs. This is not a problem for us because we know each other.
Driver updates are a bit more complicated. You need to update the drivers on the host and in all of the CTs. It’s best to not update the drivers too often.
The CTs share the same kernel with the host. To avoid potential compatibility issues, we don’t update the kernel unless necessary.

Proxmox VE on Charlie Chiang's blog

Installing Proxmox VE on an OVHcloud Kimsufi KS-5

Hardware

Why Install Proxmox Manually

Format NVMe Drives as 4K LBA

Open the IPMI KVM

Do Not Stream the Proxmox ISO Through JViewer

Boot Proxmox Through netboot.xyz

Install Proxmox on ZFS RAID1

Optional Partition Alignment Fix

Fix OVHcloud Boot-to-Disk

Basic Post-Install ZFS Settings

Reduce Unnecessary Host Writes

Configure BBR

Configure Swap

SSH and Serial Console

Scratch Storage

mdadm + LVM Alternative

Using Scratch Storage in VMs and CTs

TRIM

Proxmox Networking on a Single Public IP

NAT and Port Forwarding

DHCP and DNS with Pi-hole

Host Monitoring

Service CT

SMTP Notifications

Final State

Proxmox VE Shared GPU Installation Guide

Introduction

Proxmox VE Installation

Proxmox VE Post-Installtion

Change APT Sources

Change CT Template Sources

Stop Cluster Services

Install Common Tools

Configure Shell

Configure ZRAM Swap

Enable IPv6 SLAAC

TRIM Optimizations

Limit Journal Size

NVIDIA P2P Driver Installation

Disable IOMMU

Disable ACS

Enable Above 4G Decoding

Install Official Driver

Install P2P-Enabled Kernel Modules

Enable Persistence Mode

Build CT Templates

Install Docker

Revert Containerd Config

Add GPUs to CT

Install GPU Driver

Install CUDA

Install NCCL

Install NVIDIA Container Toolkit

Install NVIDIA Nsight Systems

Functionality Tests

NCCL

CUDA P2P

Prepare for Templating

Other Minor Settings

RAID Card Write Back

Obstacles

Only 255 Cores are Recognized - x2APIC

Share NVIDIA GPU between CTs in Proxmox VE

Background

Install Drivers on the Host

Allow NVIDIA Device Passthrough in CT

Install Drivers in CT

Missing nvidia-uvm and High Idle Power Draw

Downsides

Missing `nvidia-uvm` and High Idle Power Draw