Containerization on Charlie Chiang's blog

Proxmox VE Shared GPU Installation Guide

Sat, 16 Aug 2025 20:49:00 +0800

Introduction

Most schools and universities in China have a limited number of GPU servers (often only one or two, or even none). This makes the management of GPU development machines quite different than corporation which have virtually unlimited GPU servers.

Let’s consider a normal lab environment with 12 students, each needing a GPU server for their projects, and we have one GPU server with 8 GPUs available.

If we give each student a dedicated GPU server, we would need 12 servers, this is impractical and costly.
If we let all the students share a single GPU server (all students have access to all resources), it would be very difficult to manage, as each student would need to install their own software and dependencies, leading to conflicts. Soon the GPU server will become a hot mess. Making each student use a different user account would not solve the problem, as they would still share the same OS and software environment, and will potentially break each other’s environment.
If we make use of IOMMU and PCIe passthrough, we can assign a virtual machine with one dedicated passthrough’d GPU to each student. This approach has almost no interference between students, as each student has their own OS and software environment, even with it’s own GPU. However, this leads to a lot of wasted resources, because most of the time the GPU is idle, and each student will only have access to one GPU at a time, even if all of the GPUs are idle.
If we use GPU virtualization, we can assign a virtual machine with a shared GPU to each student. This way, each student can use the GPU resources as needed, and the GPU can be shared among multiple students. This approach is more efficient and cost-effective, as it allows for better resource utilization. However, it requires a GPU that supports virtualization, such as NVIDIA’s vGPU or AMD’s MxGPU. Most consumer GPUs do not support virtualization, so this approach is not feasible for most schools and universities.

This is where containerization comes in. By using containerization, we can create a shared GPU development server that runs a container for each student. Each container can have its own software environment, and all (or some of) of the GPUs can be shared among multiple containers. This approach is more efficient and cost-effective, as it allows for better resource utilization, and does not require a GPU that supports virtualization. However, it should be noted that this approach is not as isolated as virtual machines, as all containers share the same kernel and GPU resources. Therefore, it is important to ensure that the containers are properly configured, and that the students are aware of the limitations and potential issues that may arise from sharing the same GPU resources (for example, if one student runs a GPU-intensive task, it may affect the performance of other students’ containers).

We will be using the CT (LXC Containers) in Proxmox VE to achieve this. Why not use Docker? Because Docker is meant for running applications, not for running full Linux distros. Although you can use something like sysbox to run a full Linux distro in Docker, i would still prefer to use LXC Containers, as they are built for this purpose and are already built into Proxmox VE, making it easier to manage and deploy.

Proxmox VE Installation

Download the latest Proxmox VE ISO from the official website. At the time of writing, the latest version is Proxmox VE 9.0. Use whatever method you prefer to install Proxmox VE, such as using a USB drive.

I will use IPMI of the server to install Proxmox VE, as it is the most convenient method for me. You can also use a monitor and keyboard to install Proxmox VE if you prefer.

Mount the Proxmox VE installation ISO as virtual media so we can boot into.

Now reboot the server into the virtual media. Supermicro motherboards lets you invode the boot menu using F11. Your motherboard may have a different shortcut.

Choose our CDROM virtual media.

Boot into the Proxmox install menu (I prefer terminal UI over GUI ones).

After you accepted the license, you will need to choose the target installtion disk. You should choose the boot SSDs on your server, not data drives. The drive in picture is 3*1.92TB SSDs in RAID 5.

Toggle Advanced options. Keep ext4, we will not be using ZFS. I know ZFS has CoW, snapshotting, compressing, checksums, and a ton of other features. But we will not use ZFS, because it will potentially cause some problems, e.g., high IO load when using RAID-Z zvols, extremely slow container start times when using ZFS storage driver in Docker, SSD write amplifications, low SSD random read/write performance, ZFS ARC not being given back to the OS as fast as needed on high memory pressure systems leading to OOM, and a ton of other issues that I previously encountered. To save me some trouble, I will use the battle-tested ext4.

Total size: keep as-is
Swap size: 0, we don’t need it, our memory is large enought (1TiB) and we will later use ZRAM as swap
Maximum root volume size: 100 (GiB), make it slightly larger so we can install things into the root volume, but not too large to occupy the data volume space.

Other options can be left empty.

Other install steps (Keyboard setup, root password) can proceed as you normally would do. Remember to set a complex root password.

Regarding to IP addresses:

If you are going to use a static IP address like I do, you can just set it here and forget about it.

If you are going to use a dynamic (DHCP) addresses, you should keep what’s already in here (static IP). You don’t need to change anything. The valid IP address should already be automatically discovered from DHCP servers and filled in. After the installation, you can change the static IP mode to DHCP mode by running vi /etc/network/interfaces and make the following edits:

1
2
3
4
5
6
7
8


 auto vmbr0
- iface vmbr0 inet static
+ iface vmbr0 inet dhcp # Enable DHCP and remove static IP.
- address 10.112.154.220/16
- gateway 10.112.0.1
 bridge-ports eno1
 bridge-stp off
 bridge-fd 0

After you enabled DHCP in the config, apply it by ifreload -a. Check if it takes effect by ip a show vmbr0, you should see something like valid_lft 7160sec preferred_lft 7160sec. If there is no xx sec inside it, your DHCP is malfunctioning.

1
2
3
4
5
6
7


root@z8:/etc/network# ip a show vmbr0
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
 link/ether 30:13:8b:6d:3a:a6 brd ff:ff:ff:ff:ff:ff
 inet 10.112.154.220/16 brd 10.112.255.255 scope global dynamic vmbr0
 valid_lft 7160sec preferred_lft 7160sec
 inet6 fe80::3213:8bff:fe6d:3aa6/64 scope link
 valid_lft forever preferred_lft forever

Proxmox VE Post-Installtion

Change APT Sources

In China, we don’t really have a good global Internet connection, so we will change APT source to mirrors in China.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# Change APT source mirrors
sed -i 's/deb.debian.org/mirrors.ustc.edu.cn/g' /etc/apt/sources.list.d/debian.sources
sed -i 's/security.debian.org/mirrors.ustc.edu.cn/g' /etc/apt/sources.list.d/debian.sources

# Change Ceph source mirrors
if [ -f /etc/apt/sources.list.d/ceph.sources ]; then
 CEPH_CODENAME=`ceph -v | grep ceph | awk '{print $(NF-1)}'`
 source /etc/os-release
 cat > /etc/apt/sources.list.d/ceph.sources <<EOF
Types: deb
URIs: https://mirrors.ustc.edu.cn/proxmox/debian/ceph-$CEPH_CODENAME
Suites: $VERSION_CODENAME
Components: no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF
fi

Remove enterprise repo and add no-subscription repo.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Remove enterprise sources
rm /etc/apt/sources.list.d/pve-enterprise.sources

# Add no-subscription sources
cat > /etc/apt/sources.list.d/pve-no-subscription.sources <<EOF
Types: deb
URIs: https://mirrors.ustc.edu.cn/proxmox/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF

Change CT Template Sources

Use mirrors in China.

1
2


sed -i.bak 's|http://download.proxmox.com|https://mirrors.ustc.edu.cn/proxmox|g' /usr/share/perl5/PVE/APLInfo.pm
systemctl restart pvedaemon

Stop Cluster Services

We are not using PVE clusters. Disable them. If you are using PVE clusters, you probably will not be reading this guide :P

1
2
3


systemctl disable --now pve-ha-crm.service
systemctl disable --now pve-ha-lrm.service
systemctl disable --now corosync.service

Install Common Tools

Useful tools that will be used regularly.

1

apt install htop sysstat vim sudo

Configure Shell

I like ZSH and my dotfiles, so I will use them.

1

apt install git zsh

1
2
3
4
5
6
7
8


cd
git clone --depth=1 https://github.com/charlie0129/dotfiles.git
cd dotfiles
./bootstrap.sh -f
chsh -s /usr/bin/zsh

# Run zsh and follow the instructions
zsh

Configure ZRAM Swap

1
2
3
4
5


git clone --depth=1 https://github.com/foundObjects/zram-swap.git
cd zram-swap
./install.sh
cd ..
rm -rf zram-swap

The default lz4 is a good balance between speed and compression ratio. If you want to sacrifice speed for better compression, you can change the config to use zstd as the compression method.

1

sed -i 's/_zram_algorithm=.*/_zram_algorithm="zstd"/g' /etc/default/zram-swap

Apply

1

systemctl restart zram-swap

Tune kernel parameters to make better use of ZRAM

1
2
3
4
5
6


cat <<EOF > /etc/sysctl.d/zram.conf
vm.swappiness = 180
vm.watermark_boost_factor = 0
vm.watermark_scale_factor = 125
vm.page-cluster = 0
EOF

Apply

1

sysctl --system

Enable IPv6 SLAAC

Proxmox disables IPv6 by default. To enable IPv6 SLAAC:

1
2
3
4
5
6
7
8


cat <<EOF > /etc/sysctl.d/ipv6.conf
net.ipv6.conf.default.accept_ra = 2
net.ipv6.conf.all.accept_ra = 2
net.ipv6.conf.default.forwarding = 1
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.proxy_ndp = 1
net.ipv6.conf.all.proxy_ndp = 1
EOF

Apply it

1
2
3


sysctl --system
# Note: the following command may break your internet connection.
systemctl restart networking

TRIM Optimizations

TRIM will help prolong SSD lifespan and maintain performance. Enable it on the LVM thin pool.

1

sed -i 's/.*issue_discards = .*/\tissue_discards = 1/g' /etc/lvm/lvm.conf

Since LXCs are just on the host, it will automatically issue discards if the underlying storage supports it, so no further action is needed. If you are using VM, be sure to use SCSI disks (VirtIO SCSI Single) with discard=on option. I’ve heard that VirtIO Block devices will not work with dicards but I haven’t tested it yet. Correct me if I’m wrong.

Limit Journal Size

1

sed -i 's/.*SystemMaxUse.*/SystemMaxUse=32M/g' /etc/systemd/journald.conf

Apply

1
2


systemctl daemon-reload
systemctl restart systemd-journald

NVIDIA P2P Driver Installation

Instead of regular NVIDIA drivers, we will be installing P2P-enabled drivers to force enable PCIe P2P capabilities on consumer cards (like GeForce RTX 4090). This will bring performance boost (~10%) across multiple scenarios. For details, refer to my blog post: Enabling PCIe P2P on NVIDIA RTX 4090s

Disable IOMMU

Since PCIe P2P in Linux doesn’t work so well with IOMMU-enabled systems (there are many potential issues you may run into), you may as well just disable IOMMU. Note that if you have more than 255 CPU cores, you will only have access to 255 CPUs due to APIC fallback.

To do disable IOMMU, you can either

Disable Intel VT-d (AMD is on by default) in BIOS
Disable intel_iommu or amd_iommu in Linux kernel parameters

To disable intel_iommu or amd_iommu in Linux kernel parameters, use the following command to add intel_iommu=off and amd_iommu=off to your GRUB_CMDLINE_LINUX_DEFAULT. BTW: I also reduce the screen resolution to 1024x768 because it’s a server

1

sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=off amd_iommu=off video=1024x768@60"/g' /etc/default/grub

Reboot your system to see the effect.

1

reboot

Make sure the follow command produces NO entries. If there are output, it means IOMMU is not correctly disabled.

1

ls /sys/class/iommu

Disable ACS

You may need to disable ACS to get PCIe P2P to work, refer to https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html (PCI Access Control Services)

Enable Above 4G Decoding

PCIe P2P will need to access each GPU’s memory, so the PCIe BAR should be large enough to cover GPU’s memory. You must enable Above 4G Decoding in your motherboard settings to make the PCIe BAR large enough to work. Enable it in your motherboard’s BIOS.

Install Official Driver

Before we install the P2P kernel module, we need to install the official drivers.

Since we are installing unofficial P2P kernel modules, there is a limited number of driver version that will work. The version of the P2P kernel modules MUST be the same as the official driver we are going to install.

Refer to https://github.com/tinygrad/open-gpu-kernel-modules to see what versions are available. I also have my patches that works with much newer driver versions (575.57.08) (at the time of writing) https://github.com/charlie0129/open-gpu-kernel-modules .

I will be using version 575.57.08, so we should download the official driver of the same version (575.57.08). You should download the .run file, e.g., NVIDIA-Linux-x86_64-575.57.08.run

1
2
3


# Assume you download the installer as NVIDIA-Linux-x86_64-575.57.08.run
installer="NVIDIA-Linux-x86_64-575.57.08.run"
chmod +x $installer

Skip kernel modules because we will install P2P-patched version later.

1

./$installer --no-kernel-modules

Choose whatever you want, doesn’t matter.

Yes, disable nouveau, we will be using NVIDIA drivers.

Do not abort, just continue. Nouveau will be disabled on next boot.

Just choose the default option for every step that comes later.

Remove the driver installation file because we will not be using it later.

1

rm $installer

You can skip reboot now. We will reboot after we installed the kernel modules.

Install P2P-Enabled Kernel Modules

Make sure the version of the P2P-enabled kernel modules match the version of the driver. I will use veriosn 575.57.08.

1

git clone --depth=1 -b 575.57.08-p2p https://github.com/charlie0129/open-gpu-kernel-modules.git

To build kernel modules, install build dependencies and kernel source. sudo installed because the install script uses sudo but we don’t have it now.

1

apt install sudo build-essential proxmox-headers-$(uname -r)

Build and install the kernel module

1

./install.sh

You can safely ignore the NVIDIA-SMI failure as long as the build succeeds (you should see a DEPMOD /lib/modules/6.14.8-2-pve line) because we haven’t rebooted yet (so NVIDIA-SMI can’t be used).

Remove the source after installing

1
2


cd ..
rm -rf open-gpu-kernel-modules

Reboot your system and you should see nvidia-smi running fine.

To tell if P2P is enabled, we will use a simple method (performance testing will be done later):

1

nvidia-smi -q | grep -i bar -A 3

You should see > 2048 MiB BAR1 Total memory, 32768 MiB in my case.

1
2
3
4


 BAR1 Memory Usage
 Total : 32768 MiB
 Used : 2 MiB
 Free : 32766 MiB

Enable Persistence Mode

Persistence mode will:

Keep the GPU driver running so program can start faster
Lower GPU power mode when it’s idle to save power. For RTX 4090s, it can drop from ~70W to ~10W.
Make sure /dev/nvidia* device nodes are ready. This is useful because we are passing the GPU devices /dev/nvidia* to CTs (LXC Containers) later, it requires the device nodes to be present on start up for auto-started CTs.

So you really should enable it.

To enable persistence mode:

1

/usr/bin/nvidia-smi -pm 1

To make it persistent across reboots, we will use a cron job.

1

crontab -e

Add a line to run nvidia-smi on boot:

1

@reboot /usr/bin/nvidia-smi -pm 1 >/dev/null 2>&1

Why not use nvidia-persistenced systemd service? Because for whatever reason, it does not initialize the device nodes /dev/nvidia* correctly, so auto-started CTs will encounter Cuda failure 'unknown error'.
1
2
3
4
cd /usr/share/doc/NVIDIA_GLX-1.0/samples
tar jxf nvidia-persistenced-init.tar.bz2
cd nvidia-persistenced-init
./install.sh

You should feel nvidia-smi runs much faster than before.

Build CT Templates

We will build a CT Template that has everything a student will need:

Common build dependencies
CUDA
Docker
…

Download the Linux distro you want. I chose Ubuntu 24.04, not because I like Ubuntu (I use Debian), but because most students only know Ubuntu.

Create an unprivileged CT just like you normally would. Just remember to give it a bit more disk space (32GB or more) because we will install CUDA later and CUDA is really large.

Install Docker

After you created the CT, shut it down. Add additional settings to allow Docker to use overlayfs driver, otherwise Docker images will be extremely inefficient in CTs.

1
2
3
4
5
6
7


# This command should be run on the host, not CT.
# Assume 8000 is the ID of your template CT
cat <<EOF >> /etc/pve/local/lxc/8000.conf
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
EOF

All following commands should be run in the CT unless specified otherwise.

Add Docker configuration. Systemd journal is used as log driver to make use of previously configured journal size limit. By writing logs to a centralized location, it’s also useful if you want to write logs to memory to reduce disk writes (e.g. log2ram).

1
2
3
4
5
6
7
8


mkdir -p /etc/docker/
cat <<EOF > /etc/docker/daemon.json
{
 "live-restore": true,
 "experimental": true,
 "log-driver": "journald"
}
EOF

Install Docker

1
2


export DOWNLOAD_URL=https://mirrors.ustc.edu.cn/docker-ce
curl -fsSL https://get.docker.io | sh

Revert Containerd Config

Docker modified containerd config. We will revert it to the default config.

1

containerd config default >/etc/containerd/config.toml

Add GPUs to CT

Choose which GPU you need to passthrough by looking for the index in nvidia-smi, or by ls -l /dev/nvidia*

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# This command should be run on the host, not CT.
# ls -l /dev/nvidia*
crw-rw-rw- 1 root root 195, 0 2025-08-17 01:47:58 /dev/nvidia0
crw-rw-rw- 1 root root 195, 1 2025-08-17 01:48:00 /dev/nvidia1
crw-rw-rw- 1 root root 195, 2 2025-08-17 01:48:01 /dev/nvidia2
crw-rw-rw- 1 root root 195, 3 2025-08-17 01:48:02 /dev/nvidia3
crw-rw-rw- 1 root root 195, 4 2025-08-17 01:48:03 /dev/nvidia4
crw-rw-rw- 1 root root 195, 5 2025-08-17 01:48:05 /dev/nvidia5
crw-rw-rw- 1 root root 195, 6 2025-08-17 01:48:06 /dev/nvidia6
crw-rw-rw- 1 root root 195, 7 2025-08-17 01:48:07 /dev/nvidia7
... (omitted)

For example, if want to add all 8 GPUs (0, 1, 2, 3, 4, 5, 6, 7) to CT, I will need to run:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# This command should be run on the host, not CT.
# Assume 8000 is the ID of your template CT
cat <<EOF >> /etc/pve/local/lxc/8000.conf
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia1 dev/nvidia1 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia2 dev/nvidia2 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia3 dev/nvidia3 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia4 dev/nvidia4 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia5 dev/nvidia5 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia6 dev/nvidia6 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia7 dev/nvidia7 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/fb0 dev/fb0 none bind,optional,create=file
EOF

Note the lxc.mount.entry: /dev/nvidiaX dev/nvidiaX none bind,optional,create=file lines. Each line represents a single GPU to add to the CT. I will add 8 GPUs, so I add 8 lines from nvidia0 through nvidia7. Remove or add them as needed.

Check if the GPUs are there inside CT:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# ls -l /dev/nvidia*
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:00 /dev/nvidia-modeset
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:50:43 /dev/nvidia-uvm
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:50:43 /dev/nvidia-uvm-tools
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:47:58 /dev/nvidia0
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:00 /dev/nvidia1
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:01 /dev/nvidia2
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:02 /dev/nvidia3
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:03 /dev/nvidia4
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:05 /dev/nvidia5
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:06 /dev/nvidia6
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:07 /dev/nvidia7
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:47:58 /dev/nvidiactl

We haven’t installed drivers yet, so nvidia-smi is not available.

Install GPU Driver

Important: you should use the same driver version as the host (575.57.08 in my case).

1
2
3
4


wget <driver-runfile-url> -O driver.run
chmod +x driver.run
driver.run --no-kernel-modules
rm driver.run

Just install as normal.

Now you should be able to use nvidia-smi and see the GPUs you added to the CT (8 GPUs in my case).

Install CUDA

Important: current driver version MUST be greater than (or equal to) the driver required by CUDA. For example, CUDA 12.9.0 will work on my machine because CUDA 12.9.0 requires 575.00, and my current driver version is 575.57.08 (greater than 575.00).

1
2
3
4


wget <cuda-runfile-url> -O cuda.run
chmod +x cuda.run
cuda.run
rm cuda.run

Remember to deselect the driver (we already installed the correct driver):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


┌──────────────────────────────────────────────────────────────────────────────┐
│ CUDA Installer │
│ - [ ] Driver <== deselect this │
│ [ ] 575.51.03 <== deselect this │
│ + [X] CUDA Toolkit 12.9 │
│ [ ] CUDA Demo Suite 12.9 <== deselect this (not useful, takes up space) │
│ [ ] CUDA Documentation 12.9 <== deselect this (not useful, takes up space) │
│ - [ ] Kernel Objects │
│ [ ] nvidia-fs │
│ Options │
│ Install │
│ │
│ Up/Down: Move | Left/Right: Expand | 'Enter': Select | 'A': Advanced options │
└──────────────────────────────────────────────────────────────────────────────┘

After CUDA installation, configure PATH and LD_LIBRARY_PATH per printed instructions.

1
2
3
4
5


... (omitted)
Please make sure that
 - PATH includes /usr/local/cuda-12.9/bin
 - LD_LIBRARY_PATH includes /usr/local/cuda-12.9/lib64, or, add /usr/local/cuda-12.9/lib64 to /etc/ld.so.conf and run ldconfig as root
... (omitted)

Note that you can use /usr/local/cuda instead of /usr/local/cuda-XX.X so it’s independent of CUDA versions.

Also add export CUDA_HOME=/usr/local/cuda to your shell rc file.

nvcc should be available after a shell reload:

1
2
3
4
5
6


# nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Apr__9_19:24:57_PDT_2025
Cuda compilation tools, release 12.9, V12.9.41
Build cuda_12.9.r12.9/compiler.35813241_0

Install NCCL

You will need NCCL for most GPU-related communications.

You should find a NCCL that’s compatible with your CUDA version. For example, NCCL 2.27.3 is compatible with my CUDA version 12.9.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


# Download NCCL package
wget https://xxx/nccl_2.27.3-1+cuda12.9_x86_64.txz

tar Jxf nccl_2.27.3-1+cuda12.9_x86_64.txz

# Set some variables
NCCL_VERSION=2.27

mkdir -p /usr/local/nccl-$NCCL_VERSION

cp -vRf nccl_2.27.3-1+cuda12.9_x86_64/* /usr/local/nccl-$NCCL_VERSION
rm -rf nccl_2.27.3-1+cuda12.9_x86_64

ln -s /usr/local/nccl-$NCCL_VERSION /usr/local/nccl

echo "/usr/local/nccl/lib" >>/etc/ld.so.conf
ldconfig

ln -s /usr/local/nccl/include/nccl.h /usr/local/include/nccl.h

# Remove NCCL package
rm https://xxx/nccl_2.27.3-1+cuda12.9_x86_64.txz

Also add export NCCL_HOME=/usr/local/nccl in your shell rc file,

Install NVIDIA Container Toolkit

To run Docker containers with GPUs.

Note that the URLs below are mirrors in China. You can use original URL if you prefer.

1
2
3
4
5
6
7
8


curl -fsSL https://mirrors.ustc.edu.cn/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://mirrors.ustc.edu.cn/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
 sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
 tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sed -i 's/nvidia.github.io/mirrors.ustc.edu.cn/g' /etc/apt/sources.list.d/nvidia-container-toolkit.list

apt-get update
apt-get install -y nvidia-container-toolkit

Configure Docker and Containerd to use NVIDIA Container Toolkit

1
2


nvidia-ctk runtime configure --runtime=docker
nvidia-ctk runtime configure --runtime=containerd

Since we are running inside unprivileged containers, we need to set no-cgroups to true

1

nvidia-ctk config --set nvidia-container-cli.no-cgroups --in-place

Install NVIDIA Nsight Systems

So we can profile applications running on the GPU.

1
2
3


wget https://xxx/NsightSystems-linux-cli-public-2025.3.1.90-3582212.deb
dpkg -i NsightSystems-linux-cli-public-2025.3.1.90-3582212.deb
rm -f NsightSystems-linux-cli-public-2025.3.1.90-3582212.deb

Functionality Tests

NCCL

1
2
3
4


git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
git reset --hard 903918f # I only tested this commit.
make

P2P ON: NCCL_P2P_LEVEL=sys ./build/all_reduce_perf --minbytes 8 --maxbytes 128M --stepfactor 2 --ngpus 8 (change --ngpus accordingly)

P2P OFF: NCCL_P2P_DISABLE=1 ./build/all_reduce_perf --minbytes 8 --maxbytes 128M --stepfactor 2 --ngpus 8 (change --ngpus accordingly)

You should see a bandwidth increase after P2P is on.

CUDA P2P

1
2
3
4
5


git clone https://github.com/NVIDIA/cuda-samples.git
git reset --hard 9c688d7 # I only tested this commit.
cd cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest
make
./p2pBandwidthLatencyTest

You should see lower GPU-GPU latency, higher bandwidth if P2P is on.

Prepare for Templating

Check if there are unnecessary files and remove them.

1
2


cd
ls -l

Remove history

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


cd

unset HISTFILE
rm .*_history

echo -n >/var/log/lastlog
echo -n >/var/log/wtmp
echo -n >/var/log/btmp

journalctl --vacuum-time=1s

rm -f /var/log/*.log

Now, it’s ready for converting to a template. To give each student a new development container, just clone this template.

Other Minor Settings

RAID Card Write Back

To achieve maximum write speed, you can enable write back mode on your RAID card, so data is written to cache first. This will significantly (yes, by A LOT, sometimes over 100x) improve random write speed on HDDs (SSDs are already very fast). If your RAID card has a battery, you don’t need to worry about data integrity in case a power failure.

Write cache policy:

Write Throuh: no cache
Write Back: cached if battery backup is present on the RAID card
Always Write Back: always cached

Obstacles

Only 255 Cores are Recognized - x2APIC

I noticed that I have one CPU offline. It should recognize 256 CPUs but I only have 255 CPUs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# lscpu
Architecture: x86_64
 CPU op-mode(s): 32-bit, 64-bit
 Address sizes: 48 bits physical, 48 bits virtual
 Byte Order: Little Endian
CPU(s): 256
 On-line CPU(s) list: 0-254
 Off-line CPU(s) list: 255 <--------------- one CPU is offline -----------
Vendor ID: AuthenticAMD
 Model name: AMD EPYC 7763 64-Core Processor
 CPU family: 25
 Model: 1
 Thread(s) per core: 2
 Core(s) per socket: 64
 Socket(s): 2

After some digging, I learned that I need to enable x2APIC on both the motherboard and kernel to have more than 255 CPU cores recognized.

In BIOS Local APIC Mode should already be x2APIC .

Now I realized it can be because I disabled IOMMU for PCIe P2P before. It turns out that disableing IOMMU can limit the number of available logical cores to 255. The reason is that the Linux kernel disables x2APIC in this case and falls back to APIC, which can only enumerate a maximum of 255 (logical) cores.

I will keep IOMMU disabled because I need PCIe P2P to function (I don’t want to deal with P2P with IOMMU on). Just forget about the 256th core :p.

gVisor Killed My Node - A Debug Process

Mon, 15 Apr 2024 23:38:00 +0800

Recent Update: One month later, after we opened our issue #10264, it turns out that the gVisor developers are also aware of tha lacking of documentation about the systemd-cgroup option. Finally:

They have added the systemd-cgroup option to the documentation in 8c3abba, although the documentation website doesn’t seem to be updated as of now.

runsc will throw a warning if it detects a systemd-like path but systemd-cgroup is not used to tell you about the possible misconfiguration.

So if you are using gVisor with systemd cgroup, you should add the systemd-cgroup option to the runsc configuration, if not already.

The Problem

Recently we found some of our Kubernetes nodes were constant going down, completely dead, with no luck connecting to it. Taints like node.kubernetes.io/unreachable:NoSchedule and node.kubernetes.io/unreachable:NoExecute were automatically added to the nodes because the kubelet was not able to communicate with the API server. The only way to bring the node back was to restart it. After some debugging, we found out that some bad gVisor-created (runsc runtime) Pods was eating all the memory of the node and killing it. Traditional Pods with runc runtime were running fine.

You may say, “But there are memory limit on a Pod. If it uses more than the limit it should be killed by the OOM killer, right”？ That’s what I thought too, so I applied the following Deployment to intentionally consume some memory and see if it will be killed.

Click to see full yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
 app: memory-eater-bash
 name: memory-eater-bash
 namespace: default
spec:
 replicas: 1
 selector:
 matchLabels:
 app: memory-eater-bash
 template:
 metadata:
 labels:
 app: memory-eater-bash
 spec:
 containers:
 - command:
 - bash
 args:
 - -c
 - big_var=data; while true; do big_var="$big_var$big_var"; done; sleep 2d
 image: python:3.12-bookworm
 name: ubuntu
 securityContext:
 readOnlyRootFilesystem: true
 runAsNonRoot: true
 runAsUser: 999
 resources:
 limits:
 cpu: 100m
 memory: 256Mi
 requests:
 cpu: 100m
 memory: 200Mi
 dnsPolicy: Default
 hostNetwork: true
 restartPolicy: Always
 runtimeClassName: gvisor

I applied 256MiB memory limit on it, hoping it will be kill after it uses more than 256MiB memory. But it didn’t. It kept eating all the memory until the node was dead, bypass OOM protection provided by Kubernetes. It ate over 500GiB of memory, which is the total memory of the node, and we were no longer able to connect to the node. Luckily, we eviction process kicked in and existing Pods on the node are moved to other nodes, so the service was not affected by much.

The Debug Process

Of course, we need to apply the above memory-eater Deployment to reproduce the issue. But killing a node every time is not a good idea 😂 , especially when we need to manually restart it. So I used another Deployment to consume some (1GiB) memory, but not all of it.

Click to see full yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
 app: memory-eater-python
 name: memory-eater-python
 namespace: default
spec:
 replicas: 1
 selector:
 matchLabels:
 app: memory-eater-python
 template:
 metadata:
 labels:
 app: memory-eater-python
 spec:
 containers:
 - command:
 - python
 args:
 - -c
 - big_list = []; print('Allocating 1GB of memory...'); [big_list.append(' ' * 10**6) for _ in range(1000)]; import time; time.sleep(100000)
 image: python:3.12-bookworm
 name: ubuntu
 securityContext:
 readOnlyRootFilesystem: true
 runAsNonRoot: true
 runAsUser: 999
 resources:
 limits:
 cpu: 100m
 memory: 256Mi
 requests:
 cpu: 100m
 memory: 200Mi
 dnsPolicy: Default
 hostNetwork: true
 restartPolicy: Always
 runtimeClassName: gvisor

Just as I expected, the memory limit did not work. Time to debug.

Trying to reproduce

Before I trying to do anything on the production cluster, I tried to reproduce the issue on my local machine. Just to be safe not to kill the production cluster and get fired :P (just kidding. It’s because debugging is a lot easier on my local machine).

I used Docker with gVisor as per the official documentation. It was working fine. Our production cluster is using containerd, so was it a containerd issue? I was in a hurry so I used minikube with gVisor addon enabled, which should provide a Kubernetes cluster with containerd and gVisor (and a containerd shim for runsc). At the time of writing, it was Kubernetes v1.28.3, containerd v1.6.24, and gVisor release-20240401.0.

I know, I know. We should keep versions of the software same as the production cluster. But I was in a hurry and I was not sure if the issue was with containerd or gVisor. So I just went with it. Spoiler: yes, it has something to do with versions. If I were to do it again, I would use the same versions as the production cluster.

1

minikube start --addons="gvisor" --container-runtime=containerd --driver=docker --cpus=6

I applied the memory-eater Deployment and waited some surprising results. But the 256MiB memory limit was working as expected. The Pod was killed after it used more than 256MiB memory. I smelled some fishy things going on in the production cluster.

Debugging on the production cluster

Again, I applied the memory-eater Deployment, but on the production cluster this time. Hoping it will not kill the node. To my relief, it did not kill the node. But it did not kill the Pod with its memory (969MiB) above limits (256MiB) either (which is expected).

1
2
3


$ kubectl top pod memory-eater-python-67cc4dd4f7-sp2r6
NAME CPU(cores) MEMORY(bytes)
memory-eater-python-67cc4dd4f7-sp2r6 0m 969Mi

Before I dig anything deeper (like with some gVisor debugging tools), I checked the sandbox created by runsc (gVisor) to see if there is anything suspicious.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


# On production node
$ PID=$(ps aux | grep memory-eater-python | grep runsc-sandbox | awk '{print $2}')
$ cat /proc/$PID/cmdline | tr '\0' '\n'
runsc-sandbox
--log-format=json
--panic-log=/var/log/pods/default_memory-eater-python-74975bf59f-7fflh_1cf55082-d769-4a51-a424-2312c23eae82/gvisor_panic.log
--root=/run/containerd/runsc/k8s.io
--log=/run/containerd/io.containerd.runtime.v2.task/k8s.io/6a541285fa39d743dc20bd8b3169c4a8f5cd4d77542144fe82d3f6f7af469c2c/log.json
--log-fd=3
--panic-log-fd=4
boot
--apply-caps=false
--bundle=/run/containerd/io.containerd.runtime.v2.task/k8s.io/6a541285fa39d743dc20bd8b3169c4a8f5cd4d77542144fe82d3f6f7af469c2c
--controller-fd=10
--cpu-num=96
--io-fds=5,6
--mounts-fd=7
--overlay-mediums=0,0
--setup-root=false
--spec-fd=11
--start-sync-fd=8
--stdio-fds=12,13,14
--total-host-memory=540886331392
--total-memory=540886331392 # 503GiB
--user-log-fd=9
--product-name=AS
-4124GS-TNR
--proc-mount-sync-fd=22
6a541285fa39d743dc20bd8b3169c4a8f5cd4d77542144fe82d3f6f7af469c2c

Hmm, something caught my eye.

1
2


--total-host-memory=540886331392
--total-memory=540886331392 # 503GiB

Why is the total-memory and total-host-memory set to 503GiB? That’s the total memory of the node. I assume it should be set to the memory limit of the Pod, which is 256MiB. To verify my assumption, I checked the sandbox created by runsc on my local machine and it indeed was set to 256MiB.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


# On my local machine
$ PID=$(ps aux | grep memory-eater-python | grep runsc-sandbox | awk '{print $2}')
$ cat /proc/$PID/cmdline | tr '\0' '\n'
runsc-sandbox
--root=/run/containerd/runsc/k8s.io
--log=/run/containerd/io.containerd.runtime.v2.task/k8s.io/f11a26742777348875e6c7f182d9f8cc5c4a1c3a7f726ad5e8cb956b0c2dea96/log.json
--log-format=json
--panic-log=/var/log/pods/default_memory-eater-python-fc75975c8-vncwz_480a4dd9-22ad-4e43-9ecb-b797735e990a/gvisor_panic.log
--log-fd=3
--panic-log-fd=4
boot
--apply-caps=false
--bundle=/run/containerd/io.containerd.runtime.v2.task/k8s.io/f11a26742777348875e6c7f182d9f8cc5c4a1c3a7f726ad5e8cb956b0c2dea96
--controller-fd=10
--cpu-num=6
--dev-io-fd=-1
--gofer-mount-confs=lisafs:none,lisafs:none
--io-fds=5,6
--mounts-fd=7
--setup-root=false
--spec-fd=11
--start-sync-fd=8
--stdio-fds=12,13,14
--total-host-memory=2061119488
--total-memory=268435456 # 256MiB
--user-log-fd=9
--proc-mount-sync-fd=22
f11a26742777348875e6c7f182d9f8cc5c4a1c3a7f726ad5e8cb956b0c2dea96

It has the correct limits on my local machine.

1
2


--total-host-memory=2061119488
--total-memory=268435456 # 256MiB

So the issue was with this --total-memory argument. But why is it incorrect on the production cluster?

Checking the versions

Seeing the difference, I checked the versions on my local cluster and the production cluster.

Tool	Local	Production
Kubernetes	1.28.3	1.26.2
containerd	v1.6.24	v1.7.0-rc.1
runsc	20240401.0	20231009.0

Both are running Ubuntu Server 22.04 LTS, using systemd+cgroupsv2.

I assume the problem was with runsc (gVisor).

What’s wrong with gVisor?

Cgroups v2 support

After I know the issue is with the --total-memory flag. I found there is a similar issue reported google/gvisor #9580. The author had great explanation of the issue:

Under Kubernetes + cgroups v2 + systemd, gVisor launches all processes into the container subgroup associated with the pause container. This makes some sense given that cgroups v2 specifies that processes can only exist at leaf nodes, and the pod’s cgroup is registered as a slice (an intermediate unit which cannot have its own processes) with systemd. When the sandbox is launched gVisor needs a container subgroup and the pause container is the first to be launched. The pause container is a child of the pod cgroup and therefore inherits the limits of the parent pod cgroup, BUT the child’s controllers reflect the default max value. This in turn means that this code which reads the memory limit and cpu quota reads these as unlimited.

Let me explain a bit more. So a normal cgroup v2 hierarchy for a Pod looks like this:

1
2
3
4


/sys/fs/cgroup/kubepods.slice/
├── kubepods-burstable.slice
│ └── kubepods-burstable-pod<pod_id>.slice # has memory limit
│ └── cri-containerd-<container_id>.scope # no memory limit

The pod slice has a memory limit (cat memory.max gives xxx), but the container scope does not (cat memory.max gives max). The container scope is where the container is running. The container scope is a child of the pod slice. So the container should inherit the memory limit of the pod slice. This is how cgroups works.

But gVisor only checks the container scope and says “oh, it has no memory limit. Let’s set the total memory to max (the total memory of the node)”. And that’s why the --total-memory is set to 503GiB on the production cluster.

This issue was fixed in google/gvisor #9631. The code below is from the PR which fixed the issue (after the comments). So it checks the Pos slice (parent) if the memory limit is not set in the container slice (leaf node).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// File: runsc/cgroup/cgroup_v2.go
// Link: https://github.com/google/gvisor/blob/43c2c00c5006e7d21ce9cd431692622031891231/runsc/cgroup/cgroup_v2.go#L333-L346
// MemoryLimit returns the memory limit.
func (c *cgroupV2) MemoryLimit() (uint64, error) {
 limStr, err := getMemoryLimit(c.MakePath(""))
 if err != nil {
 return 0, err
 }
 // In cgroupv2+systemd, limits are set in the parent slice rather
 // than the leaf node. Check the parent to see if this is the case.
 if limStr == maxLimitStr {
 parentLimStr, err := getMemoryLimit(filepath.Dir(c.MakePath("")))
 if err != nil && !errors.Is(err, os.ErrNotExist) {
 return 0, err
 }
 if parentLimStr != "" {
 limStr = parentLimStr
 }
 if limStr == maxLimitStr {
 return math.MaxUint64, nil
 }
 }
 return strconv.ParseUint(limStr, 10, 64)
}

The runsc binary on our production cluster was built before the fix was merged. So I assume the issue was fixed in the runsc binary after the merge.

Systemd cgroup configuration

So if the issue was fixed in the latest release, simply upgrading the runsc binary should fix the issue. No. The issue was still there. I checked the runsc binary on the production cluster and it was built after the fix was merged. So what’s wrong?

Remember what a normal cgroup v2 hierarchy looks like in the previous section? Although the cgroups created by runc are just like that, that’s not how the ones created by runsc looks like on our production cluster. Instead, they look like this:

1
2
3
4
5
6


/sys/fs/cgroup/
├── kubepods.slice
│ ├── kubepods-burstable.slice
│ │ └── kubepods-burstable-pod<pod_id>.slice
├── system.slice
│ ├── kubepods-burstable-pod<pod_id>.slice:cri-containerd:<container_id>

Why are the container cgroups not under the pod cgroups? Why are they in system.slice? If that was the case, the fix above won’t do anything because the parent slice is not the pod slice. The parent slice is system.slice, it has no memory limit. So the container will have no memory limit even after the fix. OOM’in the node.

After some issue-searching on the gVisor GitHub repo, I found that runsc initially has no support for systemd cgroups google/gvisor #193. Support for systemd cgroup was added in google/gvisor #7287. It required a specific configuration in the runsc configuration:

1
2
3
4


// File: runsc/config/config.go
// Link: https://github.com/google/gvisor/blob/bf86207401cab99d859b25acd4911038608f0d33/runsc/config/config.go#L229-L230
+ // Use systemd to configure cgroups.
+ SystemdCgroup bool `flag:"systemd-cgroup"`

Hmm, looks like there was nothing like that in the runsc configuration on the production cluster. Go! Fix it! Find one of the cordoned node, drain it, upgrade the runsc binary, and change the configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


/etc/containerd/config.toml
 [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
 runtime_type = "io.containerd.runsc.v1"
+ [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc.options]
+ TypeUrl = "io.containerd.runsc.v1.options"
+ ConfigPath = "/etc/containerd/runsc.toml"

/etc/containerd/runsc.toml"
+ [runsc_config]
+ systemd-cgroup = "true"

Restart containerd systemctl restart containerd. Apply a new Pod with toleration and nodeSelector to schedule it to a cordoned node. And it worked! The container cgroups are now under the pod cgroups. The --total-memory is set to 256MiB. The Pod was killed after it used more than 256MiB memory. The node was not killed. The issue was fixed.

Conclusion

So the issue was a combination of old gVisor version and incorrect configuration. The gVisor version on the production cluster was built before the fix was merged. The configuration was missing the systemd-cgroup option. The issue was fixed by upgrading the runsc binary and adding the systemd-cgroup option to the configuration.

Docker and ZFS - A Tough Pair

Sun, 07 Aug 2022 08:54:00 +0800

Outline:

Know ZFS filesystem and Docker’s zfs storage driver
How Docker’s zfs driver works and Why it is slow
Why we can’t directly use overlayfs on ZFS
- Why overlayfs don’t accept remote filesystems
- Why ZFS is identified as a remote filesystem
How to actually solve the problem

We will dive into the source code of moby, OpenZFS, and Linux kernel to find out.

Note: I admit this blog is not so beginner-friendly, which requires some prerequisites, otherwise you may have a hard time reading it though. I will give some questions or concepts after each prerequisite to help you know your understanding is enough on this topic.

general computer/unix concepts (block devices, copy-on-write, mount points)
basic filesystem concepts (difference between block devices and filesystems, common filesystems, basic understanding of Linux Virtual Filesystem)
the basics of ZFS filesystem (terminologies like datasets and snapshots, what are rollbacks)
Docker images (what are image layers, when they are created/deleted, how it works with UnionFS)

Background

What is ZFS

Described as The last word in filesystems, ZFS is scalable, and includes extensive protection against data corruption, support for high storage capacities, efficient data compression, integration of the concepts of filesystem and volume management, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, RAID-Z, native NFSv4 ACLs, and can be very precisely configured. ¹

By saying ZFS, I am referring to OpenZFS on Linux and FreeBSD: OpenZFS Documentation

ZFS is a great and sophisticated filesystem, really robust and stable. It never failed my expectations. I personally use ZFS on my personal devices, whenever possible, e.g. laptops (Ubuntu Desktop - for its built-in support for ZFS), NAS (TrueNAS SCALE), and servers (Proxmox VE and Ubuntu Server).

What is a Docker storage driver

Docker uses storage drivers to store image layers, and to store data in the writable layer of a container. The container’s writable layer does not persist after the container is deleted, but is suitable for storing ephemeral data that is generated at runtime. Storage drivers are optimized for space efficiency, but (depending on the storage driver) write speeds are lower than native file system performance, especially for storage drivers that use a copy-on-write filesystem. Write-intensive applications, such as database storage, are impacted by a performance overhead, particularly if pre-existing data exists in the read-only layer. ²

By default, Docker will use overlay2 whenever possible for all Linux distributions.

ZFS and Docker storage driver?

The Docker Engine provides a zfs storage drivers on Linux, which requires a ZFS filesystem, allowing for advanced options, such as creating snapshots, but require more maintenance and setup.

According to Docker docs, the zfs storage driver has the following advantages ³ :

Avoids the container’s writable layer grow too large in write-heavy workloads.
Performs better for write-heavy workloads (though not as well as Docker volumes).
A good choice for high-density workloads such as PaaS.

Hmm, sounds good, right? Well, keep reading. If it is that good, this blog wouldn’t exist at the first place.

In this blog, zfs refers to Docker’s zfs storage driver, mostly, but may also refer to ZFS filesystem. You should be able to distinguish them by context.

What’s the problem?

There is one single problem with ZFS that bothered me since the very beginning: Docker.

When using Docker on ZFS, one can only use its zfs driver (no, you cannot use overlay2 directly. We will see why later.). Although Docker docs proudly advertise its zfs driver as something high-performance:

zfs is a good choice for high-density workloads such as PaaS. ³

But in practice, this thing is slow as hell, specifically, when creating image layers. Build times can go from a fraction of a second on overlay2 to several minutes on zfs!

It is several magnitudes slower, a complete disaster.

Let’s take the Dockerfile in kube-trigger (a project that I worked on recently) as an example.

Click to see the complete Dockerfile

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


# Copyright 2022 The KubeVela Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Note: the ${BIN} needs to be replaced with the actual binary,
# otherwise it won't work. Refer to Makefile for how it can be done.

ARG BUILD_IMAGE=golang:1.17
ARG BASE_IMAGE=gcr.io/distroless/static:nonroot

# Force native build platform, and cross-build to target platform later.
FROM --platform=${BUILDPLATFORM:-linux/amd64} ${BUILD_IMAGE} as builder

WORKDIR /workspace
COPY go.mod go.mod
COPY go.sum go.sum

ARG GOPROXY
ENV GOPROXY=${GOPROXY}
RUN go mod download

COPY build build
COPY hack hack
COPY cmd cmd
COPY api api
COPY controllers controllers
COPY pkg pkg

ARG TARGETARCH
ARG ARCH
ARG TARGETOS
ARG OS
ARG VERSION
ARG GOFLAGS
ARG DIRTY_BUILD
ARG ENTRY

RUN ARCH=${TARGETARCH:-${ARCH:-amd64}} \
 OS=${TARGETOS:-${OS:-linux}} \
 OUTPUT=${BIN} \
 VERSION=${VERSION} \
 GOFLAGS=${GOFLAGS} \
 /bin/sh build/build.sh \
 ${ENTRY}

FROM ${BASE_IMAGE}
WORKDIR /
COPY --from=builder /workspace/${BIN} .
ENTRYPOINT ["/${BIN}"]

We will focus on L39-L46:

1
2
3
4
5
6
7
8


ARG TARGETARCH
ARG ARCH
ARG TARGETOS
ARG OS
ARG VERSION
ARG GOFLAGS
ARG DIRTY_BUILD
ARG ENTRY

You might be thinking, this is just some build args, so what? Yes, this part almost does nothing (creates some image layers), and should finish immediately. That’s exactly the case on overlay2, but not on zfs, which will take minutes!

Such slow build times are driving me crazy.

Why `zfs` driver is so slow?

How ZFS storage driver works?

When using docker on a zfs dataset, the only option is Docker’s zfs driver, which uses ZFS dataset operations to create layered filesystems. The zfs storage driver for Docker stores each layer of each image as a separate legacy dataset. Even just a handful of images can result in a huge number of layers, each layer corresponding to a legacy ZFS dataset. As a result, there are hundreds of datasets created when only running a dozen containers.

The base layer of an image is a ZFS filesystem. Each child layer is a ZFS clone based on a ZFS snapshot of the layer below it. A container is a ZFS clone based on a ZFS Snapshot of the top layer of the image it’s created from. ⁴

Where’s the bottleneck?

Although when building images it do not have to deal with such many datasets. It will still spend a fair amount of time mounting and unmounting these datasets (can be seen from Docker debug logs).

We can take a look at the code from Docker daemon (moby/moby).

Mount will happen (if necessary) whenever Get is called:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


// File: daemon/graphdriver/zfs/zfs.go
// Link: https://github.com/moby/moby/blob/7e44b7cddd43b1771a44a2dd56548627e491c950/daemon/graphdriver/zfs/zfs.go#L365-L408
// Get returns the mountpoint for the given id after creating the target directories if necessary.
func (d *Driver) Get(id, mountLabel string) (_ containerfs.ContainerFS, retErr error) {
 d.locker.Lock(id)
 defer d.locker.Unlock(id)
 mountpoint := d.mountPath(id)
 if count := d.ctr.Increment(mountpoint); count > 1 {
 return containerfs.NewLocalContainerFS(mountpoint), nil
 }
 defer func() {
 if retErr != nil {
 if c := d.ctr.Decrement(mountpoint); c <= 0 {
 if mntErr := unix.Unmount(mountpoint, 0); mntErr != nil {
 logrus.WithField("storage-driver", "zfs").Errorf("Error unmounting %v: %v", mountpoint, mntErr)
 }
 if rmErr := unix.Rmdir(mountpoint); rmErr != nil && !os.IsNotExist(rmErr) {
 logrus.WithField("storage-driver", "zfs").Debugf("Failed to remove %s: %v", id, rmErr)
 }

 }
 }
 }()

 filesystem := d.zfsPath(id)
 options := label.FormatMountLabel("", mountLabel)
 logrus.WithField("storage-driver", "zfs").Debugf(`mount("%s", "%s", "%s")`, filesystem, mountpoint, options)

 root := d.idMap.RootPair()
 // Create the target directories if they don't exist
 if err := idtools.MkdirAllAndChown(mountpoint, 0755, root); err != nil {
 return nil, err
 }

 if err := mount.Mount(filesystem, mountpoint, "zfs", options); err != nil {
 return nil, errors.Wrap(err, "error creating zfs mount")
 }

 // this could be our first mount after creation of the filesystem, and the root dir may still have root
 // permissions instead of the remapped root uid:gid (if user namespaces are enabled):
 if err := root.Chown(mountpoint); err != nil {
 return nil, fmt.Errorf("error modifying zfs mountpoint (%s) directory ownership: %v", mountpoint, err)
 }

 return containerfs.NewLocalContainerFS(mountpoint), nil
}

Unmount will happen whenever Put is called:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// File: daemon/graphdriver/zfs/zfs.go
// Link: https://github.com/moby/moby/blob/7e44b7cddd43b1771a44a2dd56548627e491c950/daemon/graphdriver/zfs/zfs.go#L410-L431
// Put removes the existing mountpoint for the given id if it exists.
func (d *Driver) Put(id string) error {
 d.locker.Lock(id)
 defer d.locker.Unlock(id)
 mountpoint := d.mountPath(id)
 if count := d.ctr.Decrement(mountpoint); count > 0 {
 return nil
 }

 logger := logrus.WithField("storage-driver", "zfs")

 logger.Debugf(`unmount("%s")`, mountpoint)

 if err := unix.Unmount(mountpoint, unix.MNT_DETACH); err != nil {
 logger.Warnf("Failed to unmount %s mount %s: %v", id, mountpoint, err)
 }
 if err := unix.Rmdir(mountpoint); err != nil && !os.IsNotExist(err) {
 logger.Debugf("Failed to remove %s mount point %s: %v", id, mountpoint, err)
 }

 return nil
}

Although Docker will not mount a filesystem twice, changes still exist when consecutive Get/Put call happens.

I am not an OpenZFS developer, but it seems to me that there is a bottleneck with ZFS with such frequent mount/unmount actions (with a large mount datasets and snapshots).

As you can see, Docker is already optimizing this situation by using the mount syscall directly (instead of calling user-space mount command, which will again, after going to kernel-space from user-space, require the kernel to call the zfs mount binary in user-space, due to ZFS’s license issues with the vfs_mount in kernel).

One possible solution

So there is not much to optimize in zfs storage drivers. It is the actual zfs mount process that is slowing image build times down. Now, the problem with Docker’s zfs storage driver is clear. There are two options left:

optimize ZFS mount times
just get rid of zfs storage driver

Of course, you can also grab another disk in ext4 as use overlayfs on top of it. But I only have zfs-formatted disks, so I only have the above two options.

The first one “optimize ZFS mount times” isn’t really an option. Currently, I don’t have the expertise or time to work on OpenZFS.

With that out of the way, we only have one option left: “do not use zfs storage driver”, i.e., use overlay2 on zfs datasets.

But it doesn’t work

Now, the problem is, how do we use overlay2 storage driver on a ZFS filesystem (dataset)?

Simply put, that’s not possible (directly). ZFS makes use of d_revalidate . Having d_revalidate set to not NULL will make overlayfs refuse to work.

But why? To understand, we need to analyze some source code from OpenZFS and Linux kernel.

What’s `d_revalidate`?

d_revalidate is defined in Linux kernel include/linux/dcache.h:

1
2
3
4
5
6


// File: include/linux/dcache.h
// Link: https://github.com/torvalds/linux/blob/1612c382ffbdf1f673caec76502b1c00e6d35363/include/linux/dcache.h#L128
struct dentry_operations {
 int (*d_revalidate)(struct dentry *, unsigned int);
 // The rest are omitted.
} ____cacheline_aligned;

The kernel documentation has a nice description on d_revalidate.

TL;DR: d_revalidate is typically used with network filesystems, and is called called when the VFS needs to revalidate a dentry, marking this dentry is still valid or not, to prevent things change without the client being aware of it.

d_revalidate is called called when the VFS needs to revalidate a dentry. This is called whenever a name look-up finds a dentry in the dcache. Most local filesystems leave this as NULL, because all their dentries in the dcache are valid. Network filesystems are different since things can change on the server without the client necessarily being aware of it.

This function should return a positive value if the dentry is still valid, and zero or a negative error code if it isn’t.

d_revalidate may be called in rcu-walk mode (flags & LOOKUP_RCU). If in rcu-walk mode, the filesystem must revalidate the dentry without blocking or storing to the dentry, d_parent and d_inode should not be used without care (because they can change and, in d_inode case, even become NULL under us).

If a situation is encountered that rcu-walk cannot handle, return -ECHILD and it will be called again in ref-walk mode.

Excerpt from: Overview of the Linux Virtual File System — The Linux Kernel documentation

Why use `d_revalidate`?

ZFS makes use of d_revalidate to invalidate dcache after rolling back.

See? This is the same situation as the kernel doc describes. When a roll back happens, the underlying files are changed, but the dcache it not updated, so we need to use d_revalidate to mark it as invalid.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


// File: module/os/linux/zfs/zpl_inode.c
// Link: https://github.com/openzfs/zfs/blob/1d3ba0bf01020f5459b1c28db3979129088924c0/module/os/linux/zfs/zpl_inode.c#L701-L739
static int
#ifdef HAVE_D_REVALIDATE_NAMEIDATA
zpl_revalidate(struct dentry *dentry, struct nameidata *nd)
{
 unsigned int flags = (nd ? nd->flags : 0);
#else
zpl_revalidate(struct dentry *dentry, unsigned int flags)
{
#endif /* HAVE_D_REVALIDATE_NAMEIDATA */
 /* CSTYLED */
 zfsvfs_t *zfsvfs = dentry->d_sb->s_fs_info;
 int error;

 if (flags & LOOKUP_RCU)
 return (-ECHILD);

 /*
 * After a rollback negative dentries created before the rollback
 * time must be invalidated. Otherwise they can obscure files which
 * are only present in the rolled back dataset.
 */
 if (dentry->d_inode == NULL) {
 spin_lock(&dentry->d_lock);
 error = time_before(dentry->d_time, zfsvfs->z_rollback_time);
 spin_unlock(&dentry->d_lock);

 if (error)
 return (0);
 }

 /*
 * The dentry may reference a stale inode if a mounted file system
 * was rolled back to a point in time where the object didn't exist.
 */
 if (dentry->d_inode && ITOZ(dentry->d_inode)->z_is_stale)
 return (0);

 return (1);
}

d_revalidate is set to the zpl_revalidate function that we have seen above.

1
2
3
4
5


// File: module/os/linux/zfs/zpl_inode.c
// Link: https://github.com/openzfs/zfs/blob/1d3ba0bf01020f5459b1c28db3979129088924c0/module/os/linux/zfs/zpl_inode.c#L830-L832
dentry_operations_t zpl_dentry_operations = {
 .d_revalidate = zpl_revalidate,
};

But why is having d_revalidate set to not NULL will make overlayfs refuse to work?

How `overlayfs` refuses `d_revalidate`-enabled fs

To understand how and why How overlayfs refuses d_revalidate-enabled fs, let’s turn our focus to the Linux kernel.

`DCACHE_OP_REVALIDATE` flag is set

If a dentry has d_revalidate set to not NULL, which is the case with ZFS, kernel will mark DCACHE_OP_REVALIDATE in its d_flags. A d_flags is just flags to tell what operations that this dentry supports, and DCACHE_OP_REVALIDATE means it supports d_revalidate operations.

Now in our case, ZFS uses d_revalidate, so our d_flags have a DCACHE_OP_REVALIDATE present.

Keep this in mind. This flag will cause overlayfs to identify it as a remote fs. You will see why later.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


// File: fs/dcache.c
// Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/dcache.c#L1915-L1943
void d_set_d_op(struct dentry *dentry, const struct dentry_operations *op)
{
 WARN_ON_ONCE(dentry->d_op);
 WARN_ON_ONCE(dentry->d_flags & (DCACHE_OP_HASH |
 DCACHE_OP_COMPARE |
 DCACHE_OP_REVALIDATE |
 DCACHE_OP_WEAK_REVALIDATE |
 DCACHE_OP_DELETE |
 DCACHE_OP_REAL));
 dentry->d_op = op;
 if (!op)
 return;
 if (op->d_hash)
 dentry->d_flags |= DCACHE_OP_HASH;
 if (op->d_compare)
 dentry->d_flags |= DCACHE_OP_COMPARE;
 if (op->d_revalidate)
 dentry->d_flags |= DCACHE_OP_REVALIDATE;
 if (op->d_weak_revalidate)
 dentry->d_flags |= DCACHE_OP_WEAK_REVALIDATE;
 if (op->d_delete)
 dentry->d_flags |= DCACHE_OP_DELETE;
 if (op->d_prune)
 dentry->d_flags |= DCACHE_OP_PRUNE;
 if (op->d_real)
 dentry->d_flags |= DCACHE_OP_REAL;

}
EXPORT_SYMBOL(d_set_d_op);

Mounting process of `overlayfs`

To understand how overlayfs rejects d_revalidate enabled fs, we need to look at the code that mounts overlayfs.

When we mount a overlayfs, we call ovl_mount() in the kernel fs/overlayfs.

1
2
3
4
5
6
7


// File: fs/overlayfs/super.c
// Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/overlayfs/super.c#L2158-L2162
static struct dentry *ovl_mount(struct file_system_type *fs_type, int flags,
 const char *dev_name, void *raw_data)
{
 return mount_nodev(fs_type, flags, raw_data, ovl_fill_super);
}

ovl_fill_super() will be called to fill the dir (workdir) where overlayfs mounts, which will call ovl_get_workdir.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


// File: fs/overlayfs/super.c
// Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/overlayfs/super.c#L2047-L2079
static int ovl_fill_super(struct super_block *sb, void *data, int silent)
{
 // Omitted till L2048
 if (ofs->config.upperdir) {
 struct super_block *upper_sb;

 err = -EINVAL;
 if (!ofs->config.workdir) {
 pr_err("missing 'workdir'\n");
 goto out_err;
 }

 err = ovl_get_upper(sb, ofs, &layers[0], &upperpath);
 if (err)
 goto out_err;

 upper_sb = ovl_upper_mnt(ofs)->mnt_sb;
 if (!ovl_should_sync(ofs)) {
 ofs->errseq = errseq_sample(&upper_sb->s_wb_err);
 if (errseq_check(&upper_sb->s_wb_err, ofs->errseq)) {
 err = -EIO;
 pr_err("Cannot mount volatile when upperdir has an unseen error. Sync upperdir fs to clear state.\n");
 goto out_err;
 }
 }

 err = ovl_get_workdir(sb, ofs, &upperpath);
 if (err)
 goto out_err;

 if (!ofs->workdir)
 sb->s_flags |= SB_RDONLY;

 sb->s_stack_depth = upper_sb->s_stack_depth;
 sb->s_time_gran = upper_sb->s_time_gran;
 }
}

Saw something called lowerdir and upperdir? Here’s what they means in overlayfs:

An overlay filesystem combines two filesystems - an ‘upper’ filesystem and a ’lower’ filesystem. When a name exists in both filesystems, the object in the ‘upper’ filesystem is visible while the object in the ’lower’ filesystem is either hidden or, in the case of directories, merged with the ‘upper’ object.

It would be more correct to refer to an upper and lower ‘directory tree’ rather than ‘filesystem’ as it is quite possible for both directory trees to be in the same filesystem and there is no requirement that the root of a filesystem be given for either upper or lower.

The lower filesystem can be any filesystem supported by Linux and does not need to be writable. The lower filesystem can even be another overlayfs. The upper filesystem will normally be writable and if it is it must support the creation of trusted.* extended attributes, and must provide valid d_type in readdir responses, so NFS is not suitable.

A read-only overlay of two read-only filesystems may use any filesystem type.

Excerpt from Overlay Filesystem — The Linux Kernel documentation

And ovl_get_workdir will get where the workdir is and it call ovl_make_workdir to make the workdir.

1
2
3
4
5
6
7
8


// File: fs/overlayfs/super.c
// Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/overlayfs/super.c#L1513
static int ovl_get_workdir(struct super_block *sb, struct ovl_fs *ofs,
 struct path *upperpath)
{
 // Omitted till L1513
 err = ovl_make_workdir(sb, ofs, &workpath);
}

ZFS is identified as a remote fs

Now the good bit comes, notice the comments. This where the overlayfs rejects remote fs. (We will see why ZFS is identified as remote fs later.)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


// File: fs/overlayfs/super.c
// Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/overlayfs/super.c#L1434-L1444
static int ovl_make_workdir(struct super_block *sb, struct ovl_fs *ofs,
 struct path *workpath)
{
 // Omitted till L1433
 /*
 * We allowed sub-optimal upper fs configuration and don't want to break
 * users over kernel upgrade, but we never allowed remote upper fs, so
 * we can enforce strict requirements for remote upper fs.
 */
 if (ovl_dentry_remote(ofs->workdir) &&
 (!d_type || !rename_whiteout || ofs->noxattr)) {
 pr_err("upper fs missing required features.\n");
 err = -EINVAL;
 goto out;
 }
}

In ovl_dentry_remote, it directly marks dentry which has DCACHE_OP_REVALIDATE flags (Remember what we said before? ZFS sets this flag.) as remote, and thus the code above will go into the if-condition, then rejecting it.

1
2
3
4
5
6
7


// File: fs/overlayfs/super.c
// Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/overlayfs/util.c#L97-L101
bool ovl_dentry_remote(struct dentry *dentry)
{
 return dentry->d_flags &
 (DCACHE_OP_REVALIDATE | DCACHE_OP_WEAK_REVALIDATE);
}

Now everything comes together. Ah, this is why having d_revalidate set to not NULL will lead to Linux treating ZFS as a remote filesystem (like NFS) and thus things like overlayfs won’t work with ZFS.

There is PRs in OpenZFS to fix this problem: https://github.com/openzfs/zfs/pull/9600 , https://github.com/openzfs/zfs/pull/9414 . But currently they are held and I don’t the expertise or time to work on it either. Hopefully wish I can pick up that PR and fix it (if possible).

Final solution

So, the only option left is not possible now. Is there something we can do?

As I said earlier, “Simply put, that’s not possible (directly)”. Well, it turns out, there is still an indirect way – ZFS Volumes. Let’s see what Oracle says:

A ZFS volume is a dataset that represents a block device.

Excerpt from: https://docs.oracle.com/cd/E19253-01/819-5461/gaypf/index.html

Note that it is a block device. This is really important, which means we can treat it like a conventional hard drive and do whatever we want on ZFS datasets!

Since it is a block device, we can use it as a Swap device, iSCSI target, and in this case, a block device holding a ext4 partitation to put overlayfs on.

Solve problem

Finally! We now decide to use ZFS Volumes (zvol) to hold our overlayfs , i.e., overlayfs on top of ext4 on top of zvol on top of ZFS datasets. (Well, it is a bit complex. But trust me, even with so many fs layers, the performance is still wayyyy higher than Docker’s zfs driver.)

Let’s fix this now.

Stop Docker:

1

sudo systemctl stop docker

Destroy the dataset that Docker uses previously. You can use zfs list to find all datasets. In our case, it is rpool/ROOT/ubuntu_uzcb39/var/lib/docker.

1
2
3


# I mount this dataset at /var/lib/docker, which docker uses. Remove it.
sudo zfs destroy rpool/ROOT/ubuntu_uzcb39/var/lib/docker -R -r
# Be careful! This will destroy datasets recursively.

Create a ZFS Volume.

1
2
3
4
5


sudo zfs create -sV 64G rpool/ROOT/ubuntu_uzcb39/var/lib/docker
# rpool/ROOT/ubuntu_uzcb39/var/lib/docker is where the dataset is. Note that this will not be mounted to /var/lib/docker, which is different from the one above.
# -V creates a zvol
# -s makes it sparse, i.e, dynamically expands instead of taking all defined space
# set a max size of 64G

Format the zvol. ZFS Volumes are identified as devices in the /dev/zvol/{dsk,rdsk}/pool directory. Since we created a block device, let’s format it to ext4.

1
2


# The zvol we just created is mounted at /dev/zvol/rpool/ROOT/ubuntu_uzcb39/var/lib/docker.
sudo mkfs.ext4 /dev/zvol/rpool/ROOT/ubuntu_uzcb39/var/lib/docker

Mount the ext4 partitation to /var/lib/docker.

1
2
3


sudo mkdir -p /var/lib/docker
# Mount ext4-formatted zvol (block device) to /var/lib/docker
sudo mount /dev/zvol/rpool/ROOT/ubuntu_uzcb39/var/lib/docker /var/lib/docker

Check if it is successfully mounted.

1
2
3


df -hT
# /dev/zd0 ext4 63G 5.8G 54G 10% /var/lib/docker
# It is mounted, great!

Start Docker back up and check status.

1
2
3
4
5
6
7
8


sudo systemctl start docker
docker info
# We are using overlay2 now!
# > Storage Driver: overlay2
# > Backing Filesystem: extfs
# > Supports d_type: true
# > Native Overlay Diff: true
# > userxattr: false

Make changes persistent. Make sure the zvol is automatically mounted to /var/lib/docker after system reboots.

1
2
3


sudo vim /etc/fstab
# Append this:
# /dev/zvol/rpool/ROOT/ubuntu_uzcb39/var/lib/docker /var/lib/docker ext4 defaults 0 0

Horrey! Build time are several orders of magnitude faster now!

Containerization on Charlie Chiang's blog

Proxmox VE Shared GPU Installation Guide

Introduction

Proxmox VE Installation

Proxmox VE Post-Installtion

Change APT Sources

Change CT Template Sources

Stop Cluster Services

Install Common Tools

Configure Shell

Configure ZRAM Swap

Enable IPv6 SLAAC

TRIM Optimizations

Limit Journal Size

NVIDIA P2P Driver Installation

Disable IOMMU

Disable ACS

Enable Above 4G Decoding

Install Official Driver

Install P2P-Enabled Kernel Modules

Enable Persistence Mode

Build CT Templates

Install Docker

Revert Containerd Config

Add GPUs to CT

Install GPU Driver

Install CUDA

Install NCCL

Install NVIDIA Container Toolkit

Install NVIDIA Nsight Systems

Functionality Tests

NCCL

CUDA P2P

Prepare for Templating

Other Minor Settings

RAID Card Write Back

Obstacles

Only 255 Cores are Recognized - x2APIC

gVisor Killed My Node - A Debug Process

The Problem

The Debug Process

Trying to reproduce

Debugging on the production cluster

Checking the versions

What’s wrong with gVisor?

Cgroups v2 support

Systemd cgroup configuration

Conclusion

Docker and ZFS - A Tough Pair

Background

What is ZFS

What is a Docker storage driver

ZFS and Docker storage driver?

What’s the problem?

Why zfs driver is so slow?

How ZFS storage driver works?

Where’s the bottleneck?

One possible solution

But it doesn’t work

What’s d_revalidate?

Why use d_revalidate?

How overlayfs refuses d_revalidate-enabled fs

DCACHE_OP_REVALIDATE flag is set

Mounting process of overlayfs

ZFS is identified as a remote fs

Final solution

Solve problem

Why `zfs` driver is so slow?

What’s `d_revalidate`?

Why use `d_revalidate`?

How `overlayfs` refuses `d_revalidate`-enabled fs

`DCACHE_OP_REVALIDATE` flag is set

Mounting process of `overlayfs`