Charlie Chiang's blog

Building Alpine Linux Disk Images Without a VM

Sat, 05 Sep 2026 22:00:00 +0800

In Minimal Alpine Linux on a 1 GB Btrfs Root Disk I wrote down the recipe I use to build a tiny Alpine instance: boot the ISO in a VM, run setup-alpine with disk=none, partition by hand, setup-disk -m sys /mnt, reboot, then qemu-img convert the disk. It works, and I still use that post as the explanation of why the pieces are the way they are.

What I got tired of is the process. Every image costs a VM, a console session, and a sequence of interactive answers, and none of it is reproducible: if I want the same image with ext4 instead of btrfs, I do the whole thing again and hope I remember every step.

So I replaced it with a script. It builds the disk image as a file, on any Linux host, with no VM and no prompts. It also fixes the two things I did not like about the old recipe: the image now boots under both BIOS and UEFI, and every choice is configurable.

The bundle for this post contains everything:

mkalpine.sh — the builder
config.example.sh — every knob with comments
testboot.sh — boot the result under QEMU and check it came up
testmatrix.sh — build and boot every supported combination
hooks/ — the customizations, one file each

1

sudo ./mkalpine.sh -f alpine.img

That is the whole interface. About a minute later there is a 512 MiB sparse image, 102 MiB of it actually allocated, that boots on either firmware. The image is deliberately small — 70-growroot expands the root filesystem to fill whatever disk it lands on, so IMAGE_SIZE only has to hold the build.

Why No VM Is Needed

The manual recipe suggests that installing Alpine requires a running Alpine. It does not. Once you look at what the installer actually does, almost all of it is file manipulation:

Unpack a root filesystem.
apk add a kernel, an initramfs generator, and a bootloader.
Write /etc/fstab, /etc/inittab, and the network config.
Run mkinitfs and grub-install.
Enable some OpenRC services, which is ln -s in a directory.

Only steps 2 and 4 need to execute Alpine binaries, and a chroot is enough for that. There is no step that needs a booted kernel, a real block device, or firmware.

Alpine’s own setup-disk cannot be reused here, unfortunately. It sources libalpine.sh, calls lbu package, and assumes throughout that it is running on a booted live system. So mkalpine.sh reimplements the same steps directly, following the structure of upstream alpine-make-vm-image and the bootloader logic from setup-disk.

The pipeline is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


truncate -s 512M image create a sparse file
sfdisk partition it
losetup -P get /dev/loopNp1..p3
mkfs.vfat / mkfs.btrfs make filesystems
mount mount root, then /boot inside it
tar -x minirootfs unpack the base system
chroot + apk add kernel, mkinitfs, grub, fs tools
mkinitfs / grub-install bootloader for both firmwares
hooks/ customizations
fstrim; umount; losetup -d compact and release
qemu-img / zstd / gzip optional output formats

QEMU is not needed to build. It is used only for qcow2 output and by the boot tests.

One line of that pipeline is less innocent than it looks. losetup -P asks the kernel to scan the partition table, but the /dev/loopNp* nodes are created by udev, after losetup has already exited, so they have to be waited for — and on a host with no udev at all, such as a container whose /dev is a plain tmpfs, they have to be created by hand from the device numbers the kernel publishes in /sys/block/loopN/loopNpM/dev. Both paths are in the script, and the interesting part is the order they are tried in: reaching for mknod early is a mistake, because when udev gets round to the same events it unlinks your node and makes its own, and for the instant in between the path does not exist. On a test host, that instant landed exactly on mkfs.vfat:

1

mkfs.vfat: unable to open /dev/loop0p2: No such file or directory

So the escalation is now slow on purpose — udevadm settle, a second, partx -a, another second, and only then mknod — and once the nodes do appear there is one more settle before anything opens them, so a replacement in flight finishes before mkfs runs rather than during it.

The Disk Layout

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


GPT (default; the protective MBR still carries GRUB's boot.img)

 # size type mount contents
 1 1 MiB ef02 BIOS boot - grub core.img (x86_64 only)
 2 64 MiB ef00 ESP FAT32 /boot vmlinuz-virt, initramfs-virt,
 grub/, EFI/BOOT/BOOTX64.EFI
 3 rest L Linux / btrfs (default), ext4 or xfs

BIOS : firmware -> MBR boot.img -> p1 core.img -> /boot/grub/grub.cfg
UEFI : firmware -> p2 /EFI/BOOT/BOOTX64.EFI -> /boot/grub/grub.cfg

Two details are worth calling out.

/boot is the ESP. This is not just to save a partition. It means GRUB only ever has to read FAT. Btrfs, ext4 and XFS roots all work without GRUB parsing them at all, and no on-disk feature the root filesystem gains later can break the bootloader. The old post used Syslinux on ext4 /boot; this is less fragile.

GRUB is installed to the removable path. grub-install --removable --no-nvram writes EFI/BOOT/BOOTX64.EFI (or BOOTAA64.EFI), which is what firmware boots when NVRAM has no entry for the disk. We cannot write the target machine’s NVRAM from a build host anyway, and cloud firmware is generally starting from a blank slate.

On aarch64 the BIOS partition is omitted and only UEFI is set up: Alpine ships grub-efi for arm64 but there is no grub-bios, because the i386-pc target is x86-only.

The MBR Variant

Some providers’ image import still rejects GPT. PARTITION_TABLE=mbr handles that:

1
2
3
4
5
6
7


DOS/MBR

 # start type mount contents
 - sector 0 MBR + gap - grub boot.img (sector 0) and
 core.img (the ~1 MiB gap before p1)
 1 2048 0xEF, bootable /boot same as above
 2 after p1 0x83 / root

No dedicated BIOS boot partition is needed, because GRUB’s i386-pc target embeds core.img in the gap between the MBR and the first partition — which is about 1 MiB, given the 2048-sector start. UEFI still works on most firmware, since the ESP is located by partition type 0xEF. That is a widely followed convention rather than something the spec promises, so MBR mode is “BIOS guaranteed, UEFI very likely”, and GPT stays the default.

What It Trusts

The bootstrap is a single alpine-minirootfs tarball, resolved from latest-releases.yaml for the branch:

TLS gets the file from the mirror.
Alpine publishes a .sha256 sidecar next to every release artifact. The build fails and deletes the download on a mismatch.
Everything after that is apk, with Alpine’s signing keys, from the tarball’s own keyring.

I use the minirootfs rather than a pinned apk.static deliberately: a hardcoded apk.static checksum drifts out of sync with the branch, and the in-image apk always matching the branch matters now that 3.23+ ships apk-tools 3.

ALPINE_BRANCH=latest-stable is the default and gets resolved once, at build time. The repositories written into the image always point at the concrete branch (v3.24), never at latest-stable, so a later apk upgrade on the running machine does not silently jump to the next stable release.

Configuration

Copy the example and edit it; ./config.sh is picked up automatically.

1
2
3


cp config.example.sh config.sh
$EDITOR config.sh
sudo ./mkalpine.sh

Every variable is : "${VAR:=default}" in the script, so the environment works too, which is what I use for one-offs:

1

sudo IMAGE_SIZE=2G ROOT_FS=xfs ./mkalpine.sh out.img

Secrets default to nothing. ROOT_PASSWORD_HASH="" locks the root account, so the failure mode of forgetting to set it is “cannot log in”, not “known root password”, and the example hash ships commented out next to the openssl passwd -6 that generates one. config.sh and the image files are in .gitignore.

If the build host reaches the internet through a proxy, an exported http_proxy / https_proxy / no_proxy is picked up as-is, and BUILD_HTTP_PROXY / BUILD_HTTPS_PROXY / BUILD_NO_PROXY set one for the build alone. It covers both halves of the download: the minirootfs tarball on the host, and apk plus whatever a hook fetches inside the chroot. Both cases of each name are set in the chroot’s environment, because curl reads only the lowercase http_proxy while apk and git take either — and they go into the environment rather than a profile script, so nothing about the proxy survives into the image. A proxy reachable from the build host is usually not reachable from wherever the image is deployed, and the URL frequently carries credentials.

Choosing A Root Filesystem

ROOT_FS takes btrfs, ext4 or xfs, each with its own mkfs and mount option strings:

`ROOT_FS`	`ROOT_MKFS_OPTS`	`ROOT_MOUNT_OPTS`
`btrfs`	`-L alpine-root -K`	`rw,noatime,compress=zstd:3,ssd,discard=async,space_cache=v2`
`ext4`	`-L alpine-root -m 1 -E nodiscard`	`rw,noatime,commit=60`
`xfs`	`-L alpine-root`	`rw,noatime,logbsize=256k`

-K and -E nodiscard skip the discard pass at mkfs time, which is pointless on a sparse file and only produces confusing errors.

Btrfs with zstd stays the default for the same reason as in the old post — on a small disk, transparent compression is worth a lot.

Tuning mkfs For The Storage Underneath

The point of exposing ROOT_MKFS_OPTS is that a cloud disk is rarely a plain disk. config.example.sh carries these examples.

Aligning to a 16 KiB ZFS zvol (volblocksize=16k):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# ext4: the block size cannot exceed the kernel page size (4 KiB on x86_64),
# so -b 16384 produces a filesystem that will not mount. Align with
# stride/stripe_width instead: stride = 16K / 4K = 4.
ROOT_MKFS_OPTS="-L alpine-root -m 1 -E nodiscard,stride=4,stripe_width=4"

# or, if you really do want 16 KiB allocation clusters:
ROOT_MKFS_OPTS="-L alpine-root -m 1 -O bigalloc -C 16384"

# xfs: takes the stripe unit directly.
ROOT_MKFS_OPTS="-L alpine-root -d su=16k,sw=1"

# btrfs: nodesize is already 16K; set the sector size explicitly if the host
# page size differs from the target's.
ROOT_MKFS_OPTS="-L alpine-root -K -s 4096"

Aligning to RAID6 with 6 data disks and a 128 KiB chunk:

1
2
3
4
5
6


# ext4: stride = chunk / block = 128K / 4K = 32
# stripe_width = stride * data disks = 32 * 4 = 128
ROOT_MKFS_OPTS="-L alpine-root -m 1 -E nodiscard,stride=32,stripe_width=128"

# xfs: su = chunk, sw = number of data disks
ROOT_MKFS_OPTS="-L alpine-root -d su=128k,sw=4"

Tuning Mount Options

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# Favour throughput over sync latency. You lose more recent writes on an
# unclean shutdown, which is fine for a rebuildable instance and not fine
# for a database.
ROOT_MOUNT_OPTS="rw,noatime,commit=60,data=writeback" # ext4
ROOT_MOUNT_OPTS="rw,noatime,logbsize=256k,allocsize=1m" # xfs
ROOT_MOUNT_OPTS="rw,noatime,commit=120,compress=zstd:1,ssd,discard=async,space_cache=v2"

# Favour density on a tiny disk. zstd:6 costs CPU on write; decompression
# stays cheap at any level.
ROOT_MOUNT_OPTS="rw,noatime,compress=zstd:6,ssd,discard=async,space_cache=v2"

# Drop discard=async and rely on the weekly fstrim job instead, if your
# provider's thin pool behaves badly with continuous discards.
ROOT_MOUNT_OPTS="rw,noatime,compress=zstd:3,ssd,space_cache=v2"

These options are used to mount the image during the build as well as being written to /etc/fstab, so a typo fails the build instead of the first boot. There is exactly one deliberate difference between the two, and it is the subject of Compression Needs Forcing At Build Time below.

The Mount Option Gotcha

This one cost me a while, and it is the reason the boot test in the bundle checks mount options rather than trusting them.

ROOT_MOUNT_OPTS in /etc/fstab is not enough. The root filesystem is mounted by the initramfs, long before /etc/fstab exists, and the mount -o remount,rw / that OpenRC does afterwards cannot change every option. Most are remountable, so noatime and commit=60 looked fine. XFS, however, fixes the log buffer size at initial mount:

1
2
3
4
5


# what I asked for
ROOT_MOUNT_OPTS="rw,noatime,logbsize=256k"

# what the running machine actually had
rw,noatime,inode64,logbufs=8,logbsize=32k,noquota

Silently downgraded to the 32k default, with nothing in dmesg about it. Mounting the same image on the build host with the same options honoured logbsize=256k, which made it look like a kernel difference rather than what it was.

The fix is to put the options on the kernel command line too, so that the initial mount is the one I want:

1
2
3


ROOTFLAGS=$(echo "$FSTAB_ROOT_OPTS" | tr ',' '\n' |
 grep -vx -e rw -e ro | tr '\n' ',' | sed 's/,*$//')
[ -n "$ROOTFLAGS" ] && CMDLINE="$CMDLINE rootflags=$ROOTFLAGS"

rw and ro are dropped because the kernel handles those separately and GRUB already passes ro. With Btrfs you end up with two rootflags= on the cmdline, because grub-mkconfig’s 10_linux detects the subvolume and emits its own; ours is appended after GRUB’s and the initramfs takes the last one.

Now:

1

rw,noatime,inode64,logbufs=8,logbsize=256k,noquota

Btrfs Subvolumes

BTRFS_SUBVOL="@" by default. It costs three lines at build time, and retrofitting a root subvolume later means moving every file, so it is worth doing even on a machine too small to keep many snapshots. Compression is also set with btrfs property set, so it holds regardless of mount options. BTRFS_SUBVOL="" gives the top-level layout from the old post. GRUB does not care either way, because /boot is FAT.

Compression Needs Forcing At Build Time

compress=zstd:3 does not compress everything. Btrfs decides per file, from the beginning of the file, whether compressing is worth it — and on ELF binaries it decides wrong. In a default build, 45 MiB out of 83 MiB was stored uncompressed, including libcrypto.so.3 (3.3 MiB), every grub-* tool, busybox and ld-musl. All of those compress to roughly half. btrfs property set does not help here either: a file carrying only the inode flag still goes through the same heuristic.

compress-force skips the decision. The build mounts the root with it, while /etc/fstab and rootflags= keep plain compress=:

1
2


BUILD_ROOT_OPTS=$(printf '%s' "$ROOT_MOUNT_OPTS" |
 sed 's/^compress=/compress-force=/; s/,compress=/,compress-force=/')

That is the one place where what the build mounts deliberately differs from what the image ships. It takes 62 MiB of data on disk down to 56 MiB, the raw image from 108 MiB to 102 MiB, and df on the booted machine from 74.5 MiB used to 67.9 MiB; BTRFS_FORCE_COMPRESS=no restores the old behaviour. The build prints what it achieved —

1

root data 55M on disk for 86M of files (64%)

— because the answer depends on what the image installs, and the failure mode is otherwise silent: the mount options, and findmnt, look exactly the same whether the files got compressed or not.

The heuristic is left alone for the running system on purpose. It exists to avoid burning CPU on data that will not compress, and this image contains about 10 MiB of exactly that: Alpine ships kernel modules as .ko.gz, and xfs.ko.gz, btrfs.ko.gz and friends stay uncompressed whether you force it or not. Forcing only recovers the files the heuristic misjudged.

Running btrfs filesystem defragment -r -czstd over the finished tree is the other way to get here, and it is the worse one. It lands at 58 MiB rather than 56 MiB, the sparse image balloons to 168 MiB before fstrim claws it back, and despite the name it does not defragment anything — compressed extents cap at 128 KiB, so the extent count went up, 1996 → 2082.

One caveat if you ship a compressed artifact rather than the raw image: compressing inside the image makes the outer compressor’s job harder, and the outer compressor is better at it. Forcing shrank the raw image by 5.6% and grew every compressed output — raw.zst 74 → 77 MiB, raw.xz 72 → 76 MiB. If upload size is the thing you actually care about, BTRFS_FORCE_COMPRESS=no is the right setting, and the build says as much when you ask for both.

Sizing `/boot`

BOOT_SIZE=64M. Measured usage with one kernel is 36 MiB:

1
2
3
4
5
6


13 M vmlinuz-virt
9.4M initramfs-virt
8.3M grub/ (i386-pc and x86_64-efi modules)
6.2M System.map-6.18.48-0-virt
156K EFI/BOOT/BOOTX64.EFI
149K config-6.18.48-0-virt

That leaves headroom for one kernel and no more. Raise it to 128M if you want to keep a second kernel around — linux-lts alongside linux-virt, say — or if you want the previous kernel to survive an apk upgrade so there is something to fall back to. The build prints /boot usage at the end and warns when it lands above 80%.

Output Formats

OUTPUT_FORMATS is a space-separated list; each entry produces one file next to the raw image.

Entry	Produced by	Size	Note
`raw`	the build itself	102 MiB	sparse; 512 MiB apparent
`qcow2`	`qemu-img convert -c -O qcow2`	81 MiB	compressed qcow2
`raw.zst`	`zstd -19 -T0`	77 MiB	best ratio for the time; widely accepted
`raw.gz`	`gzip -9`	80 MiB	widest provider support
`raw.xz`	`xz -9 -T0`	76 MiB	smallest, slowest

The sizes are from one build of the same default btrfs image, so they are comparable to each other rather than absolute. Note how little the four compressed formats differ — 76 to 81 MiB, under 7% between the best and the worst: the root filesystem is already zstd-compressed, so the outer compressor is mostly squeezing free space. raw.gz is a perfectly reasonable default in exchange for its compatibility.

That is also why BTRFS_FORCE_COMPRESS cuts the other way here. Every one of these numbers except qcow2 is larger than it would be with forcing off — raw.xz most of all, 76 MiB against 72 MiB — because data that btrfs already compressed at zstd:3 in 128 KiB blocks is data xz -9 cannot compress again. Forcing wins on raw and loses on everything else, so the build prints a reminder when you ask for both.

Drop raw from the list to keep only the compressed artifacts. For other hypervisors, convert the raw image yourself:

1
2
3


qemu-img convert -O vmdk out.img out.vmdk # VMware
qemu-img convert -O vpc out.img out.vhd # Hyper-V / Azure
qemu-img convert -O vdi out.img out.vdi # VirtualBox

What Must Not Survive Cloning

A disk image is a template that gets cloned N times, so anything unique baked into it stops being unique. This is the part a hand-built VM image usually gets wrong, and it is why I wanted a script in the first place — I am not going to remember all of this at 1am on a provider’s web console.

Thing	Why it matters	What the script does
`/etc/ssh/ssh_host_*`	Every VM built from the image shares one host key. Anyone holding the image can impersonate all of them, and clients get key-mismatch warnings after the first deploy.	Delete. Regenerated on first boot.
`/var/lib/seedrng/seed.credential` (3.17+; `/var/lib/random-seed` before)	The saved RNG seed is restored early at boot. An identical seed on every clone means the entropy pool starts identical — including for the host keys in the row above.	Delete.
`/etc/machine-id`	Used to derive DHCP DUIDs and app-level instance identity, so clones can collide on DHCP leases.	Truncate to zero bytes, not delete: an empty file is the documented “generate on next boot” signal, while a missing one makes some tools fail instead.
`/etc/resolv.conf`	Would otherwise ship the build host’s nameservers.	Rewritten from `$DNS`.
`/var/cache/apk/`, `/var/log/`, shell history	Dead weight and build-host leakage.	Cleared.

Filesystem UUIDs do stay identical across clones. Regenerating them on first boot means rewriting fstab and grub.cfg from a running system, which is more fragile than the problem it solves — it only bites if you attach two clones to the same host.

The 80-firstboot hook is the other half of this. It runs in the boot runlevel, before networking and sshd, and generates what was stripped:

1
2


head -c 16 /dev/urandom | od -An -tx1 | tr -d ' \n' >/etc/machine-id
ssh-keygen -A

It also runs after hostname, so the host keys are commented root@alpine rather than root@(none). Cosmetic, but the alternative bothered me.

Finally, the build stamps /etc/image-release with the Alpine version, architecture, build date, builder git commit, and the resolved filesystem and hook configuration. Six months later, working out which build a running VM came from is otherwise guesswork.

Staying Able To Get In

The single most valuable property of a cloud image is that you can get into it when networking is broken, because the provider’s serial console is then your only channel.

console=ttyS0,115200 on x86_64, console=ttyAMA0,115200 on aarch64, with the getty in /etc/inittab and the tty listed in /etc/securetty. Without the securetty entry, root login on that console is refused and looks exactly like a wrong password.
GRUB_TERMINAL="console serial" plus GRUB_SERIAL_COMMAND, so the bootloader is reachable over serial too. That is what lets you fix a bad kernel cmdline or boot an older kernel remotely.
No quiet. Alpine’s setup-disk defaults to KERNELOPTS=quiet; on a machine whose only debugging channel is the serial console, hiding the boot log is the wrong trade.
GRUB_TIMEOUT=1 rather than 0, so there is a window to interrupt.

grub.cfg is generated by grub-mkconfig rather than hand-written, which matters more than it looks: Alpine’s grub package carries triggers="grub.trigger=/boot", so a later apk upgrade linux-virt regenerates it and the image stays bootable with no intervention. A static config would just be silently overwritten by that same trigger. There is a hand-written fallback for the case where grub-probe cannot cope with the loop device, but I have not needed it.

Hooks

One ordered list, in HOOKS. Remove a name to disable it; drop a file into hooks/ and add its name to extend. Each hook is a standalone sh script run inside the chroot with the configuration exported into its environment.

1

HOOKS="10-network 20-ssh 30-chrony 40-zram 50-logtruncate 60-sysctl 70-growroot 80-firstboot"

Hook	Does
`10-network`	`/etc/network/interfaces`, DHCP or static, optional real DHCPv6
`20-ssh`	`PermitRootLogin`, port, keepalives, `authorized_keys`
`30-chrony`	chrony with `makestep 1.0 -1` and a configurable pool
`40-zram`	zram swap sized from RAM at boot, optionally `/tmp` too
`50-logtruncate`	the hourly log cap from the old post, plus a daily apk cache clean
`60-sysctl`	zram VM tunables, BBR and fq
`70-growroot`	one-shot service: `growpart`, then grow the filesystem
`80-firstboot`	machine-id and SSH host key generation

Off by default, because they are opinionated rather than essential: 90-tools (bash, coreutils, iproute2, tcpdump, htop, tmux, vim and friends — about 120 MiB), 91-ufw, 92-sshguard, 93-podman, 94-cloud-init, and 95-dotfiles (git, zsh, lsd and a dotfiles repo, with zsh as root’s login shell).

91-ufw, 92-sshguard and 95-dotfiles are cheap enough to add at the default IMAGE_SIZE on any filesystem. The heavier three are where the compression default starts paying for itself: 90-tools + 93-podman + 94-cloud-init together fit on btrfs at 512 MiB — 274 MiB allocated — and run the 381 MiB XFS root out of space partway through installing podman. Raise IMAGE_SIZE if you want them on ext4 or XFS.

A few implementation notes:

20-ssh rewrites existing lines rather than appending. sshd_config takes the first occurrence of a keyword, so appending PermitRootLogin prohibit-password to a file that already contains a commented-out default works, and appending it to one that has an active setting silently does nothing. The hook edits in place.

50-logtruncate truncates rather than renames. Keeping one .0 copy and then truncate -s 0 on the original preserves the inode, so daemons holding the file open keep writing to it. This is the script from the old post, unchanged.

70-growroot uses growpart, not sfdisk. Growing a GPT disk also requires relocating the backup header to the new end of the device, and growpart keeps the partition’s start sector untouched, so whatever alignment the image was built with survives. After that it is btrfs filesystem resize max /, resize2fs or xfs_growfs depending on ROOT_FS, then a stamp file and rc-update del growroot default.

91-ufw exploits the fact that ufw only touches netfilter when enabled. All the rules are recorded inside the chroot with ufw itself, and then ENABLED=yes is set in ufw.conf as the last step. No netfilter calls happen during the build.

95-dotfiles is the only hook that needs the network for something other than the Alpine mirror. It shallow-clones DOTFILES_REPO, runs the repo’s own bootstrap.sh, and switches root’s login shell. Two parts of that were more interesting than expected.

Changing the login shell, first: chsh is not in a minimal Alpine — it lives in shadow, which nothing else here wants — so the hook rewrites field 7 of root’s /etc/passwd line directly. Writing the new file elsewhere and cat-ing it back keeps the original inode, mode and owner, which a mv would not:

1
2
3


awk -F: -v OFS=: -v shell="$login_shell" \
 '$1 == "root" { $7 = shell } { print }' /etc/passwd >/tmp/passwd.new
cat /tmp/passwd.new >/etc/passwd

Second, my dotfiles use zsh4humans, which installs itself — plugins, plus prebuilt fzf and gitstatusd binaries — the first time an interactive zsh starts. Doing that during the build instead buys two things: the shell works on a machine with no internet, and a download failure fails the build rather than the first login.

It costs about 67 MiB of files — 26 MiB of packages, 5 MiB of checkout, 28 MiB of z4h cache — which is 21 MiB of actual disk on the compressed btrfs default. 99-selftest checks it the way that matters: it starts an interactive zsh on the booted machine and confirms the repo’s .zshrc really was sourced.

Being the only hook that talks to GitHub also makes it the one most likely to fail a build. Set http_proxy and https_proxy when the path to GitHub is reliably bad rather than occasionally bad.

Weekly fstrim is installed regardless of ROOT_FS. Although a btrfs root already trims itself through discard=async, the job is not all about the root filesystem. A data disk attached to the instance later is quite likely to be ext4 or XFS, and nothing else in the image would ever trim it. For those, periodic trim is the currently recommended approach over -o discard.

The job also cannot be a one-liner, because a minimal Alpine image does not have util-linux:

1
2
3
4
5
6


fstrim -a 2>/dev/null && exit 0

awk '$1 ~ /^\/dev\// && !seen[$2]++ { print $2 }' /proc/mounts |
while IFS= read -r mp; do
 fstrim "$mp" 2>/dev/null
done

fstrim -a walks every mounted filesystem and de-duplicates devices, but that is util-linux’s fstrim. Busybox’s applet takes exactly one mount point and has no -a at all, so on the image as built the -a form fails and the loop does the work. My original fstrim -a || fstrim / looked like it handled that and did not: the fallback trimmed only the root, which is precisely the filesystem that needed it least.

Writing Your Own

Drop a script in hooks/, add its name to HOOKS. The whole hooks/ directory is copied into the chroot at /tmp/mkalpine, so a hook that needs to install a longer file can keep it in hooks/files/ and copy it from /tmp/mkalpine/files/ rather than embedding it in a heredoc:

1
2
3
4
5


#!/bin/sh
set -eu
apk add --quiet --no-progress my-thing
install -m 0644 /tmp/mkalpine/files/my-thing.conf /etc/my-thing.conf
rc-update add my-thing default

The number prefixes are a label, not a mechanism. This is not run-parts: nothing sorts the directory, and the run order is simply the order of the HOOKS list, so HOOKS="10-network my-thing 80-firstboot" does exactly what it looks like. Your hook can be called anything, go anywhere in the list, and it does not matter that the shipped extras have crowded the 90s — 99-selftest runs last because it is last in the list, not because of its number.

Two things about the environment a hook runs in. Everything in config.sh is exported into it, but nothing else is: the chroot starts from env -i, so a hook sees PATH, HOME, TERM, the config, and a proxy if one is configured. And it has no controlling terminal and /dev/null on stdin, for the reasons in 95-dotfiles above, so a hook cannot stop a build to ask a question — a program that tries gets EOF and fails, which is a build that ends with an error rather than one that waits all night.

Verifying The Image

An image that builds is not an image that boots, and an image that boots is not an image that is configured the way you asked. So there are two scripts.

Does It Boot

1
2


./testboot.sh -m uefi alpine.img
./testboot.sh -m bios alpine.img

This boots the image headless under QEMU with -snapshot, so the image is never modified, captures the serial console to a file, and waits for a login: prompt. Reaching the login prompt exercises the whole chain in one go: firmware → GRUB → kernel → initramfs → root mount → OpenRC → getty. On failure it prints the last 40 lines of the console, which is usually enough to see where it stopped. -w drops the -snapshot and lets the guest write to the image, for when you want to inspect what a first-boot service did to the disk.

It finds OVMF and AAVMF across the various paths different distributions use, and uses KVM when /dev/kvm is writable. With KVM, BIOS reaches the login prompt in about 16 seconds and UEFI in about 26 — OVMF initialization is the difference.

Is It Configured Correctly

Add the 99-selftest hook and pass -d:

1
2


HOOKS="$HOOKS 99-selftest" sudo ./mkalpine.sh -f alpine.img
./testboot.sh -d alpine.img

The hook installs a service that runs last in the default runlevel and checks the running system against the configuration it was built from. The expectations are baked in at build time, which is the part that makes it useful: a hook that silently did nothing shows up as a failure rather than as a plausible-looking dump.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


===== SELFTEST BEGIN =====
-- identity
 info machine-id=2767df9cefbfd2d2af2d0b8acdfb4aa5
 info hostkey=SHA256:U242JZNiEf+Qo3swIK+MoW4m2bdNb2eCMG5W6VJAwPM (ED25519)
 ok hostname is alpine
 ok machine-id is 32 hex digits
 ok timezone is UTC
-- filesystems
 info root options=rw,noatime,inode64,logbufs=8,logbsize=256k,noquota
 ok root is xfs
 ok /boot is vfat
 ok root mounted with logbsize=256k
 -- btrfs subvolume (not configured)
 ok fstab references root by UUID
-- boot
 info kernel=6.18.48-0-virt
 info cmdline=BOOT_IMAGE=/vmlinuz-virt root=UUID=e95aee24-... ro
 modules=sd-mod,usb-storage,xfs rootfstype=xfs
 rootflags=noatime,logbsize=256k console=tty0 console=ttyS0,115200
 ok cmdline is not quiet
 ok ttyS0 in securetty
-- hooks
 ok 20-ssh: PermitRootLogin is prohibit-password
 ok 20-ssh: listening on 22
 ok 40-zram: swap is ~100% of RAM
 ok 60-sysctl: congestion control is bbr
 ok 70-growroot: removed itself from the default runlevel
 ok 70-growroot: root fs fills the partition
 ok 80-firstboot: host key is not the build host's
 -- 91-ufw (disabled)
-- hygiene
 ok apk cache is empty
 ok no build scratch left behind
 ok repositories pinned to v3.24
 ok repositories do not track latest-stable
-- sizing
 Filesystem Size Used Available Use% Mounted on
 /dev/vda3 381.0M 161.2M 219.8M 42% /
 /dev/vda2 63.0M 36.4M 26.6M 58% /boot
SELFTEST RESULT: 59 passed, 0 failed
===== SELFTEST END =====

(abridged; the real report is one line per check)

testboot.sh -d exits non-zero if any check failed, so it works in a loop.

The checks live in hooks/files/selftest.initd as an ordinary shell script rather than a heredoc, and check takes a command with its arguments instead of a string to eval:

1
2
3
4


check "root is $EXPECT_ROOT_FS" fstype_is / "$EXPECT_ROOT_FS"
check "machine-id is populated" test -s /etc/machine-id
check "cmdline is not quiet" not grep -qw quiet /proc/cmdline
check "60-sysctl: vm.swappiness is 180" output_is 180 sysctl -n vm.swappiness

Anything needing a pipeline gets a named predicate instead. The first version of this used eval on quoted strings and was unreadable — check "root is btrfs" "awk '\$2 == \"/\" {print \$3}' /proc/mounts | grep -qx btrfs" — which is a good sign that the abstraction was wrong.

Everything At Once

1

sudo ./testmatrix.sh -o /var/tmp/mx

This builds btrfs/ext4/xfs × gpt/mbr, boot-tests all six under both BIOS and UEFI with the selftest enabled, then checks the things that only show up at runtime:

Growth. Copy an image, truncate -s 4G, and boot it — with testboot.sh -w, so the guest’s writes actually land in the file instead of in QEMU’s -snapshot overlay. Then confirm the root partition grew (911,360 → 8,253,407 sectors) and that it still starts on its original sector, because a growpart that moved the start would silently discard the build-time alignment. Reading the table back from the un-booted copy would have made this test tautological, which is what it was until I looked closely.
Output formats. Build with all five, confirm each file exists, and boot the qcow2 directly rather than just checking that qemu-img produced something.
Identity. Boot the same image twice and confirm the two boots produce different SSH host key fingerprints and machine-ids.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


== build and boot matrix ==
ok build btrfs/gpt
ok boot btrfs/gpt/uefi
ok boot btrfs/gpt/bios
...
== growroot on a larger disk ==
ok grow 4G
ok grow root partition 1959936 -> 8253407 sectors, still at 135168
== output formats ==
ok format formats.img.zst (74M)
ok boot qcow2
== identity is per-boot, not per-image ==
ok hygiene two boots produced different SSH host keys
ok hygiene two boots produced different machine-ids
== summary ==
29 passed, 0 failed

Four real bugs in this post were found this way. The XFS logbsize downgrade above was one, and the tautological growth check was another. The third was in the selftest itself: I had written

1

check "apk cache is empty" not ls -A /var/cache/apk

which reads correctly and means the opposite, because ls -A exits 0 on an empty directory. A test suite that catches bugs in its own checks is doing its job.

The last one showed up when I dropped the default IMAGE_SIZE from 1 GiB to 512 MiB. All three filesystems still built and booted, but ext4 and XFS started failing 70-growroot: root fs fills the partition, which had been written as “df size is at least 90% of the partition”. Nothing was wrong with the images: df excludes reserved blocks and fixed metadata, and on a 445 MiB root partition that fixed cost is 44 MiB for ext4 and 64 MiB for XFS — 90.0% and 85.6%. The same filesystems on a 957 MiB partition reported 96.5% and 93.3% and passed. A percentage was simply the wrong shape for the check, because the overhead it has to tolerate barely moves with the disk size while the threshold does. The failure it exists to catch is not marginal either — a root that never grew is a 445 MiB filesystem in a 4 GiB partition — so the tolerance is now the larger of 96 MiB and 10%.

Uploading

Most providers want a compressed raw image or a qcow2:

1

sudo OUTPUT_FORMATS="raw.zst" ./mkalpine.sh -f alpine.img

If your provider has no image import at all — which is the common case on the cheap tiers — the restore-over-dd trick from the old post still applies: boot their stock OS, dd the image onto the disk from a rescue environment or over SSH, and reboot. 70-growroot then handles the fact that their disk is larger than the image.

That flow needs a console or a rescue system at one point or another. When the provider offers neither — no VNC, no serial, no rescue mode, just the stock OS and SSH, which is what the smallest NAT’d tiers look like — use the no-console variant in the old post instead: reinstall dd mode arranges a RAM-resident Alpine by rewriting the stock bootloader, then writes the image over the whole disk and reboots, all over SSH. It accepts the raw.gz / raw.zst / raw.xz artifacts from OUTPUT_FORMATS as-is, and it does not modify a Linux image — which, on a headless box, promotes 10-network and 20-ssh from conveniences to the difference between a machine that comes back and a reinstall ticket.

What I Kept From The Old Post

Everything about why. The old post explains the 64 MiB /boot, the compressed Btrfs root, the absence of disk swap, and the individual tweaks in far more detail than a config file comment can, and it is still the reference for doing this by hand on a machine you have already booted. This post is the automation of it.

Building Audio Insight: An Audio Analyzer That Actually Renders Smoothly

Sun, 16 Aug 2026 16:42:45 +0800

I like audio analyzers. It answers questions that my ears alone cannot answer quickly. Where is that resonance? Is the low end actually mono? How loud is this over the whole track? Is a limiter catching an occasional peak, or working all the time?

What I do not like is an analyzer whose interface feels slower than the display it runs on.

On my M1 Max, Excite Audio VISION 4X appeared to top out at roughly 30 FPS, with inconsistent frame timing, while consuming about one CPU core. iZotope Insight 2 looked smoother, but in my experience it was comparatively resource-heavy and expensive. These were observations from my own setup, not controlled benchmarks that apply to every machine, host, and plugin version. Still, they were enough to make me wonder: how difficult would it be to build the analyzer I wanted?

That became Audio Insight, an open-source AUv2 and VST3 analyzer for macOS. Its first goal is deliberately narrow: show useful measurements, leave the audio unchanged, keep real-time callback work bounded, and make the interface feel native on a high-refresh-rate display.

The project is still early, but it already has a Spectrum, Spectrogram, Peak/RMS meter, stereo vectorscope and correlation meter, and BS.1770 loudness measurements. Four grid-snapped splitters resize the dashboard tiles, the analysis parameters are adjustable, and a built-in metrics panel makes the renderer’s behavior visible instead of leaving performance to intuition.

This post is about how it works, but mostly about the unexpectedly interesting work required to make a meter move smoothly.

What an audio plugin actually does

People who use plugins often picture them as little applications inside a DAW. That is a useful mental model for the interface, but not for the audio path.

An AU or VST3 plugin is code loaded by a host (or, in some hosts, a separate hosting service). The host repeatedly gives the plugin a small block of samples by calling its processing function. At 48 kHz with 512-sample blocks, a new block arrives about every 10.7 milliseconds. The plugin has to finish before the hardware needs the result. Missing that deadline can produce a click or dropout.

Audio Insight is a transparent effect: it observes supported mono or stereo audio and leaves the samples unchanged. Even so, its callback has to follow the same real-time rules as a compressor or synthesizer. It cannot allocate memory, take a lock, wait for another thread, write a log, open a file, call the UI, or ask the GPU to draw something. Any of those operations can take an unpredictable amount of time.

The resulting design looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


host audio callback
 ↓
bounded, non-blocking capture
 ↓
per-instance analysis coordinator
 ↓
shared two-worker analysis pool
 ↓
immutable measurement snapshots
 ↓
display-linked Metal renderer

The callback only captures bounded chunks into preallocated storage and updates the few measurements that must inspect every sample. A per-instance coordinator coalesces work, and all instances loaded in the same plugin module share two analysis workers. There is at most one running or queued job per instance, so opening many plugin windows does not create a thread for every visualization.

The workers publish immutable snapshots. The UI reads the newest complete snapshot whenever it draws; it never waits for analysis to finish. This separation matters. Analysis targets 60 slices per second, but the latest-wins scheduler can skip stale work rather than build a backlog. Meanwhile, a ProMotion display can render at around 120 Hz. The renderer can advance display motion between discrete analysis updates without running twice as many FFTs.

All five visualizations share one Metal canvas, drawable, command buffer, and render pass. JUCE supplies the plugin shell and cross-format plumbing, while the visual layer is native Metal. Coordinates and layout use logical points, and drawable and text resources follow the current backing scale. The implementation is therefore designed to support both regular-density and Retina displays, including live backing-scale changes.

Both paths are highly optimized. The shared transform uses juce::dsp::FFT, which selects Apple’s Accelerate/vDSP implementation on macOS, and Spectrum and Spectrogram reuse each calibrated result. On the GPU side, the Spectrogram stores calibrated dB in a circular 16-bit-float (R16Float) texture: scrolling remaps texture coordinates, while shader controls recolor retained history without another FFT or a whole-texture copy. High display cadence therefore does not multiply the default 60 Hz FFT workload.

When the editor is closed, there is nothing to display, so capture, analysis, history, display-link callbacks, and Metal submissions stop. Audio still passes through normally. Reopening the editor begins fresh rather than silently spending host resources on invisible history.

Turning samples into pictures

The analyzers share infrastructure, but each one answers a different question. Here is the calculation path in a little more detail.

Spectrum: what frequencies exist now?

The Spectrum takes a short window of recent audio and uses a fast Fourier transform (FFT) to divide it into frequency bins. By default the transform size is $N=8192$ samples, or about 171 milliseconds at a sample rate $F_s=48$ kHz. This does not delay the audio by 171 ms; the plugin passes audio through immediately. It means the displayed estimate describes roughly that much recent history.

The FFT bin centers are separated by $\Delta f=F_s/N$. With the defaults, that is approximately 5.86 Hz. This number is useful, but it is not the same as saying two tones 5.86 Hz apart can always be resolved: the selected window also determines how widely a tone spreads into nearby bins.

Before the FFT, samples are multiplied by a periodic five-term flat-top window $w[n]$. Cutting an arbitrary piece from a continuous waveform creates artificial edges, which spread energy across the spectrum. A window tapers the data to control that leakage. A flat-top window trades some ability to separate nearby tones for better amplitude accuracy, which is a useful default for a measurement tool.

For channel $c$, the transform is:

\[ \begin{aligned} X_c[k]&=\sum_{n=0}^{N-1}x_c[n]\,w[n]e^{-j2\pi kn/N},\\ f_k&=\frac{kF_s}{N}. \end{aligned} \]

Audio Insight corrects the window’s coherent gain—the amplitude scaling introduced by multiplying by $w[n]$—with $W=\sum_n w[n]$. Real-valued audio has mirrored positive- and negative-frequency FFT bins, but the graph needs only the nonnegative half. In this one-sided view, the DC bin at 0 Hz and the Nyquist bin at $F_s/2$ use $1/W$; every bin between them represents both mirrored sides and uses $2/W$. The calibrated stereo power and level are therefore:

\[ \begin{aligned} a_k&= \begin{cases} 1/W, & k=0\ \text{or}\ k=N/2,\\ 2/W, & \text{otherwise}, \end{cases}\\[3pt] P[k]&=\max_c\left(a_k|X_c[k]|\right)^2,\\ D[k]&=10\log_{10}P[k]. \end{aligned} \]

For mono, the maximum contains only one channel. For stereo, taking the larger channel magnitude avoids first mixing the waveforms to mono, where out-of-phase content could cancel. The calibration makes a bin-centered full-scale sine read 0 dB internally; powers at or below $10^{-18}$ are displayed at the $-180$ dB analysis floor.

Attack and Release then smooth each bin in linear power, not in dB. Given the elapsed time $\Delta t$ and the selected time constant $\tau_d$:

\[ \begin{aligned} \alpha_d&= \begin{cases} 0, & d\text{ is Off},\\ e^{-\Delta t/\tau_d}, & d\text{ is enabled}, \end{cases}\\[3pt] \bar P_t[k]&=\alpha_d\bar P_{t-1}[k]+(1-\alpha_d)P_t[k]. \end{aligned} \]

The direction $d$ is Attack when $P_t[k]\geq\bar P_{t-1}[k]$, otherwise Release. An Off direction follows the current FFT immediately. The default Attack is Off, allowing a short burst to appear at once, while the default 250 ms Release lets the trace fall more slowly. Peak hold, when enabled, operates on unsmoothed power instead of $\bar P$.

Transforms target a slice rate $R_s$ using a hop of $H=\max(1,\operatorname{round}(F_s/R_s))$ new samples. At 48 kHz and 60 slices per second, $H=800$, so adjacent 8,192-sample windows overlap by about 90.2%. The first result still waits for one complete window, and the latest-wins scheduler may skip stale transforms under load instead of building a backlog.

Spectrum and Spectrogram use the same continuously adjustable frequency scale. For a frequency $f$ between $f_0$ and $f_1$, the scale control $s$ blends normalized linear and logarithmic coordinates:

\[ \begin{aligned} u_{\mathrm{lin}}(f)&=\frac{f-f_0}{f_1-f_0},\\ u_{\log}(f)&=\frac{\ln(f/f_0)}{\ln(f_1/f_0)},\\ u(f,s)&=(1-s)u_{\mathrm{lin}}(f)+s\,u_{\log}(f). \end{aligned} \]

Here $f_0=20$ Hz and $f_1=\min(20\text{ kHz},F_s/2)$. The default is $s=0.8$. At $s=0$, equal distances represent equal numbers of hertz. At $s=1$, equal ratios such as 100→200 Hz and 1→2 kHz occupy equal distances. Values in between preserve more low-frequency detail without compressing the entire treble into a tiny area. Spectrum uses $x=u$, while Spectrogram uses $y=1-u$ so high frequencies appear at the top. Axis labels are chosen dynamically: important anchors win first, then extra candidates fill only the space that remains.

Spectrogram: how did the spectrum change?

A Spectrum is one slice through time. A Spectrogram keeps those slices and scrolls them sideways, using color for level. Transients become vertical marks, steady tones become horizontal lines, and harmonics become stacks of related lines.

Each Spectrogram column starts from the same raw power $P[k]$ as Spectrum, before Spectrum’s Attack/Release averaging. Let $\mathcal K$ contain only usable bin centers from 20 Hz through $f_1$, and let $R_f=\min(1024,|\mathcal K|)$ be the texture’s frequency-row count. For a usable bin at $f_k=kF_s/N$, define $q_k=u(f_k,s)$. Its row is:

\[ r(k)=\min\left(R_f-1,\left\lfloor R_f q_k\right\rfloor\right). \]

For a row containing one or more bin centers, $P_r$ is the greatest $P[k]$ assigned to that row. Taking the maximum, rather than the average, helps a narrow tonal trace survive when several FFT bins land in one display row. If a row contains no bin center—common at low frequencies with a small FFT—the mapper inverse-maps the row center and linearly interpolates the two surrounding bins in power. It is honest interpolation between available samples, not a claim of extra frequency resolution.

Power at or below $10^{-18}$ becomes $-180$ dB; otherwise the mapper stores $D_r=10\log_{10}P_r$. These values go into a circular Metal texture with one 16-bit floating-point level per cell (R16Float). The texture stores calibrated dB rather than finished colors. In the shader, let $F$ be the selected floor, $C$ the ceiling, and $\eta$ the Color response:

\[ \begin{aligned} v&=\operatorname{clamp}\left( \frac{D_r-F}{C-F},0,1\right),\\ \gamma&=2^\eta,\\ c_{\mathrm{palette}}&=v^\gamma. \end{aligned} \]

The value $c_{\mathrm{palette}}$ selects a point in the chosen palette. Response 0 is linear in dB; negative values reveal quieter detail, while positive values suppress low energy and emphasize stronger traces. Because this work happens in the shader, changing palette, range, or response recolors existing history without rerunning the FFT.

For a history duration $T$ and requested slice rate $R_s$, the texture uses $\min(8192,\lceil TR_s\rceil)$ columns. The default ten seconds at 60 slices per second therefore needs 600 columns. A write index wraps around the texture, and the renderer changes texture coordinates instead of copying the whole image to scroll it. Missing timestamp intervals become black columns rather than stretching old information across time.

Peak and RMS: how strong is the signal?

Peak and RMS intentionally describe different things.

Sample peak examines every sample. Its live value has instantaneous attack and a 20 dB/s release:

\[ \begin{aligned} \lambda_p&=10^{-20/(20F_s)},\\ p[n]&=\max\left(|x[n]|,\lambda_p p[n-1]\right). \end{aligned} \]

In other words, a new larger sample wins immediately; otherwise the old indication decays by the amount corresponding to one sample period. A separate hold marker keeps a new maximum for two seconds, then falls at the same 20 dB/s rate. The OVER indicator latches when $|x[n]|\geq1$, although the label deliberately does not claim that floating-point audio at 0 dBFS proves waveform clipping.

RMS estimates sustained signal power. With the 300 ms time constant $\tau=0.300$ s, Audio Insight updates an exponential mean square for every sample:

\[ \begin{aligned} \alpha&=e^{-1/(F_s\tau)},\\ q[n]&=\alpha q[n-1]+(1-\alpha)x[n]^2,\\ \operatorname{RMS}[n]&=\sqrt{q[n]},\\ D_{\mathrm{RMS}}[n]&=20\log_{10}\operatorname{RMS}[n]. \end{aligned} \]

This is an exponential response, not a rectangular box containing exactly the latest 300 ms. It also has no AES17 $+3.01$ dB calibration offset, so a full-scale sine reads approximately $-3.01$ dBFS RMS. Peak reveals brief extremes; RMS behaves more like a view of sustained energy. The peak remains a sample peak, not an oversampled true-peak/dBTP measurement, so it does not predict a possibly larger value between stored samples.

These ballistics run on the bounded real-time capture path and inspect every sample. Their meaning therefore does not change if an analysis worker is briefly late.

The vectorscope turns each stereo sample pair into a point:

\[ x_{\mathrm{scope}}=\frac{R-L}{2}, \qquad y_{\mathrm{scope}}=\frac{L+R}{2}. \]

Audio shared equally by both channels has $x=0$ and lies on the vertical center axis. Opposite-phase audio has $y=0$ and spreads horizontally. The coordinates remain tied to full scale rather than being normalized independently on every frame, so a quiet signal is not made to look artificially loud.

The field keeps the latest 250 ms but bounds its GPU data. For $W_f=\lceil0.25F_s\rceil$ captured frames, the worker selects one pair every:

\[ d=\left\lceil\frac{W_f}{4096}\right\rceil \]

frames. At 48 kHz, $W_f=12000$, $d=3$, and the cloud contains about 4,000 uniformly spaced points. Their opacity fades with age. This decimation changes only the picture; it does not change the correlation measurement.

The adjacent correlation value uses every sample, with 300 ms exponentially weighted averages:

\[ \begin{aligned} \alpha&=e^{-1/(F_s\cdot0.300)},\\ E_n[z]&=\alpha E_{n-1}[z]+(1-\alpha)z[n]. \end{aligned} \]\[ \rho[n]=\frac{E_n[LR]}{\sqrt{E_n[L^2]E_n[R^2]}}. \]

The three running values $E[L^2]$, $E[R^2]$, and $E[LR]$ advance in source-sample order on the real-time side, and the implementation clamps the final ratio to $[-1,1]$ against numerical error. A value near $+1$ means the channels are strongly alike, $0$ means little linear relationship, and a negative value warns that mono playback may cancel important content. If either averaged channel power is below $10^{-9}$, equivalent to $-90$ dBFS RMS, correlation is reported as unavailable rather than dividing by a nearly zero value. A genuinely mono input is labeled MONO and likewise does not receive a synthetic $+1$ correlation.

Loudness: how loud does it feel over time?

Raw peak level is not perceived loudness. Audio Insight implements BS.1770-5 K-weighting with the Momentary, Short-term, and Integrated semantics commonly used with EBU R128. K-weighting is a pair of filters: a high-frequency shelf models the head’s acoustic effect, and a high-pass stage reduces the contribution of very low frequencies. The code derives their coefficients for the current sample rate.

If $y_c[n]$ is the K-weighted output of channel $c$, Audio Insight forms the per-sample energy and a window mean:

\[ \begin{aligned} e[n]&=\sum_c y_c[n]^2,\\ z_W&=\frac{1}{N_W}\sum_{n\in W}e[n]. \end{aligned} \]

For the supported mono and stereo layouts, every actual channel has unit weight. Mono therefore contributes once; it is never duplicated into synthetic left and right channels. Surround layouts and their channel weights are outside the current scope. Window energy becomes LUFS (Loudness Units relative to Full Scale) using the BS.1770 offset:

\[ L_W=-0.691+10\log_{10}z_W. \]

Momentary loudness covers 400 ms.
Short-term loudness covers 3 seconds.
Integrated loudness uses 400 ms blocks completed every 100 ms—75% overlap—from the latest Reset within the current uninterrupted, visible analysis interval.

Momentary and Short-term are simple ungated window measurements. Integrated loudness applies gates: thresholds that exclude blocks from the long-term average. For each 400 ms block $i$, let its mean-square energy be $z_i$ and its loudness be $L_i=-0.691+10\log_{10}z_i$. The absolute-passing set is:

\[ \begin{aligned} \mathcal A&=\{i\mid L_i>-70\ \mathrm{LUFS}\},\\ \mu_{\mathcal A}&=\frac{1}{|\mathcal A|}\sum_{i\in\mathcal A}z_i. \end{aligned} \]

At each Integrated update, one non-iterative relative threshold is calculated 10 LU below the preliminary absolute-gated mean of the history accumulated so far:

\[ \begin{aligned} \Gamma_{\mathrm{rel}}&=-0.691+10\log_{10}\mu_{\mathcal A}-10,\\ \mathcal R&=\{i\in\mathcal A\mid L_i>\Gamma_{\mathrm{rel}}\}. \end{aligned} \]

Finally:

\[ \begin{aligned} \bar z_{\mathcal R}&=\frac{1}{|\mathcal R|}\sum_{i\in\mathcal R}z_i,\\ L_I&=-0.691+10\log_{10}\bar z_{\mathcal R}. \end{aligned} \]

Both comparisons are strict $>$, and the relative gate is not iterated repeatedly. This two-stage gate prevents silence and very quiet passages from dragging the program average down indefinitely. The tile’s Reset command restarts Integrated loudness while ready Momentary/Short-term values and K-weighting continuity remain intact. Editor reactivation, an audio discontinuity, or a format change resets the complete loudness analyzer.

The empty cases are explicit too. If $\mathcal A$ contains no blocks, the preliminary mean and relative gate remain unavailable. If $\mathcal R$ is empty, Integrated loudness remains $-\infty$. The implementation never divides by an empty set.

The implementation does not claim complete EBU Mode compliance: it does not yet include LRA or true peak, for example. The label describes its M/S/I measurement semantics, not a certification.

There is an interesting performance problem hiding in Integrated loudness. Within one uninterrupted visible measurement, the exact answer can cover 24 hours: up to 864,000 blocks. Rescanning every qualifying block every 100 ms would make the cost grow throughout the measurement.

The implementation uses a preallocated sorted index called a B+ tree. It contains finite block energies above the absolute gate and keeps aggregate counts and sums in its branches; all completed blocks still count toward the 24-hour limit. A new relative-gate boundary can be answered by finding one boundary leaf and combining a bounded number of branch totals. Capacity for the worst case occupies about 7.25 MiB on arm64, and the structure never allocates while processing.

Smooth is a timing property, not an FPS number

The renderer uses CAMetalDisplayLink, which supplies a drawable in step with a display. While visible, Audio Insight requests the active display’s exact reported maximum refresh rate, with a 60 Hz fallback. That request is best effort—Core Animation and the compositor still control actual presentation—so measured presentation timestamps are the truth.

This distinction became important repeatedly. A counter can say 120 callbacks per second while the screen still changes only 60 times. An average can say 120 FPS while an occasional doubled interval makes scrolling visibly hitch. Smoothness is about the complete chain from callback to presentation and about the distribution of frame intervals, not just one large number.

The plugin that crashed its host

The first AU build appeared for a moment in SoundSource and then disappeared. The host reported only that its Audio Unit hosting service had crashed.

The detailed log led to an assertion in timed drawable presentation. A normal Metal application may call an API such as timed present, but a drawable delivered by CAMetalDisplayLink has different presentation ownership. Combining the two caused the hosting process to assert. The correct sequence is to commit the command buffer and call plain present() on that drawable, while using the display link’s target timestamp only for telemetry and scheduling.

This is one reason plugin development needs testing in real hosts. SoundSource exposed an API misuse that a successful build or unit test had not.

Why 120 display callbacks produced 60 frames

After the crash was fixed, the display link was firing close to 120 times per second, but only about 60 frames were submitted. The built-in metrics captured the pattern:

Counter	Before the fix
Display-link callbacks	6,453
Metal submissions	3,230
GPU-backpressure drops	3,223
Sampled display-link callback rate	~111/s
Sampled Metal submission rate	~59.6/s

The cumulative counters cover the full telemetry epoch; the two rates are a sample from its final roughly 0.25 seconds. Almost exactly every other display-link callback was being rejected by the in-flight buffer pool.

The surprising part was that the GPU was not necessarily too slow. Reusable vertex buffers were retained until the drawable was actually presented. The compositor may hold a drawable for several refresh periods even after GPU execution has completed, so all reusable buffers became occupied and the next callback had nowhere to write.

The fix was to separate two lifetimes. GPU command completion now releases the reusable buffers immediately. A small, independent object survives only to correlate the later presentation timestamp. The renderer no longer holds large working resources hostage to compositor timing.

In a later point-in-time M1 Max capture from another development build, with the Metrics panel visible, the drawable was 2,400×1,496 pixels at 2× backing scale. The run recorded 1,188 display-link callbacks, 1,188 submissions, and zero GPU-backpressure drops. Across the most recent 240 presented intervals, the average was 8.438 ms, or 118.52 Hz; 237 intervals were the normal 8.333 ms and three doubled to 16.667 ms. Telemetry also counted 11 skipped presentations over the run. That capture used the then-selected 16,384-point FFT; today’s default is 8,192.

The capture demonstrates that the buffer-lifetime bottleneck and its GPU-backpressure drops were gone. It is evidence of approximately display-rate presentation in that run, not a perfect-pacing claim or a controlled comparison with another plugin.

Making 60 Hz data scroll on a 120 Hz display

The Spectrogram exposed a second kind of stutter. New analysis columns arrive at 60 Hz. If the image moves forward only when a complete column arrives, it necessarily steps every other frame on a 120 Hz display.

The solution was not to double the FFT workload. The renderer advances a fractional scroll head from the target presentation clock while keeping the actual dB cells discrete. A one-slice cushion absorbs ordinary analysis scheduling jitter. If a texture upload is briefly busy, the renderer postpones that upload while continuing to draw the rest of the dashboard.

The result is much smoother motion from the same 60-slices-per-second target. This also explains why raising thread priority would have been the wrong first response: the main issue was the relationship between two clocks, not a shortage of real-time privileges.

The random resets that were not random

During longer sessions, all graphs would occasionally reset. The recovery was intentional—when audio history has a real gap, temporal analyzers must not pretend the samples on either side were adjacent—but the handoff overflowed far too easily even while host audio was continuous. Sequence tracking then correctly detected the resulting loss.

The original capture queue had 16 logical slots and consumed one for each host callback. Its time capacity therefore depended on the host’s block size. A metrics capture reached all 16 ready slots, discarded 20 queued chunks to make room for newer audio, and recorded three consumer discontinuities followed by three Loudness resets.

The redesigned queue packs audio across callback boundaries into 128 slots of 256 frames, retaining 32,768 frames regardless of host callback size. That is about 683 ms at 48 kHz, 341 ms at 96 kHz, or 171 ms at 192 kHz. The capacity and overflow behavior are covered by implementation tests; longer post-redesign host runs remain part of validation. A sufficiently long stall can still overflow it. When that happens, latest data wins and temporal analyzers reset, because joining unrelated pieces of audio would produce convincing but false measurements.

Building observability into the plugin

Apple’s Metal HUD is useful for applications that enable it before creating their first Metal device. A plugin usually arrives after its host has already done that, so it cannot reliably switch the HUD on from a settings button. I replaced that idea with a built-in performance panel available in Release builds.

It reports exact frame pacing over the latest 240 presentation intervals, derived from 241 timestamps; CPU, submit, GPU, and compositor latency composition; display-link scheduling; audio-callback histograms; queue occupancy and discontinuities; analyzer freshness; and raw copyable metrics for offline inspection. The stacked latency bar covers the pipeline from a display-link callback to presentation. It is not a breakdown of an 8.33 ms frame budget: several frames can overlap in flight, so its total can exceed one refresh interval without reducing presentation cadence.

Graphs move at vblank, headline numbers refresh at no more than 10 Hz, and the full text table refreshes at 4 Hz. That keeps the visual feedback immediate without rebuilding lots of text 120 times per second. Instrumentation turned several vague reports—“it looks a bit laggy,” “it seems to reset”—into specific, actionable failures.

An agent-assisted, human-tested development loop

I used coding agents to implement much of Audio Insight. That did not remove the need for a tight feedback loop; it made the loop more important.

The agents could design the threading model, inspect crash logs, add instrumentation, and reason from raw captures. They could build the plugin, but they could not reliably judge how motion felt inside my particular SoundSource setup. At runnable milestones I installed the AU, watched it on the M1 Max, adjusted settings, and returned observations, screenshots, logs, or copied metrics. Those reports led directly to the presentation-lifetime fix, the fractional Spectrogram scroll, dynamic axis labeling, and the queue redesign.

For visual and real-time software, “the code is correct” and “the product feels right” are different claims. An instrumented implementation plus a person looking at the actual display proved far more useful than guessing at either one in isolation.

What is open source today

The current code identifies itself as Audio Insight 0.1.0. It targets macOS 15 on arm64 and builds AUv2 and VST3. It uses C++20, CMake, a pinned JUCE submodule for the plugin shell, CPU FFT analysis accelerated by Apple’s vDSP on macOS, and a native Metal renderer. Project-owned code is licensed under AGPL-3.0-or-later; JUCE retains its own upstream AGPL terms.

The current release policy is pragmatic for a small open-source project: builds use ad hoc signing, and Developer ID signing and notarization are out of scope. Users can build from source. For a downloaded bundle, the documented flow is to verify the published checksum, extract it, clear quarantine only on the intended bundle, apply an ad hoc signature, and verify that signature.

Older macOS versions, Intel/Universal builds, Windows, and AUv3 are architectural possibilities rather than current support promises. Logic compatibility, broader VST3 host coverage, multi-instance stress testing, and several formal performance gates also remain work in progress.

The source, build instructions, and current limitations are all in the Audio Insight repository. If you use analyzers but have never looked inside one, I hope the code makes the path from samples to pixels a little less mysterious. And if the Spectrogram glides across a 120 Hz display without drawing attention to the renderer, that is exactly the point.

Installing Proxmox VE on an OVHcloud Kimsufi KS-5

Mon, 06 Jul 2026 14:30:00 +0800

I ordered an OVHcloud Kimsufi KS-5 dedicated server for 19.90 USD/month. It is not fast by modern standards, but it is a very usable small Proxmox box if you install it carefully.

The goal for this machine was:

Proxmox VE on mirrored ZFS root.
Keep only part of each SSD for the root pool.
Use the remaining SSD space as a fast disposable scratch pool.
Put VMs and CTs behind NAT, because the server only has one usable public IPv4 address and one /128 IPv6 address.
Keep host writes low where it is easy to do so.

This post is written from the actual KS-5 I installed. The finished machine is running Proxmox VE 9.2.

Hardware

The server I received:

Part	Value
Product	Kimsufi KS-5
CPU	Intel Xeon E3-1270 v6, 4C/8T, 3.8 GHz base, 4.2 GHz turbo
RAM	32 GB DDR4 ECC 2400 MHz
Storage	2x Intel P3520 450 GB NVMe SSD
NIC	2x Intel I210 gigabit

The PCI devices looked like this:

1
2
3
4
5
6
7
8


00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers (rev 05)
00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31)
00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)
02:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 02)
03:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 02)
04:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1) (rev 05)
05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

The SSDs were not new in power-on hours, but the wear was low and both drives had no media errors:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# nvme smart-log /dev/nvme0n1
critical_warning : 0
temperature : 23 C
available_spare : 98%
percentage_used : 6%
Data Units Read : 8.41 TB
Data Units Written : 95.99 TB
power_cycles : 107
power_on_hours : 45628
unsafe_shutdowns : 4
media_errors : 0
num_err_log_entries : 0

# nvme smart-log /dev/nvme1n1
critical_warning : 0
temperature : 23 C
available_spare : 98%
percentage_used : 7%
Data Units Read : 142.27 TB
Data Units Written : 112.67 TB
power_cycles : 74
power_on_hours : 54153
unsafe_shutdowns : 5
media_errors : 0
num_err_log_entries : 0

For a cheap dedicated server, that is acceptable.

Why Install Proxmox Manually

OVHcloud can install an OS for you from the control panel. I did not use that path for Proxmox.

The problem is storage layout. Proxmox works best when it owns the disks directly and can put ZFS on the raw devices. The OVHcloud installer tends to build layouts around mdadm RAID and regular filesystems, or it uses ZFS in a way that treats VM/CT storage more like ordinary directories. That leaves some Proxmox storage features unavailable or awkward, especially thin-provisioned disks and the normal ZFS-backed workflow.

So I installed Proxmox myself.

Format NVMe Drives as 4K LBA

Before installing the OS, boot the server into OVHcloud rescue mode and check the NVMe namespace formats.

My Intel P3520 drives supported both 512-byte and 4096-byte LBA formats:

1
2
3
4
5
6
7
8


# nvme id-ns -H /dev/nvme0n1
LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
LBA Format 1 : Metadata Size: 8 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good
LBA Format 2 : Metadata Size: 16 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good
LBA Format 3 : Metadata Size: 0 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best
LBA Format 4 : Metadata Size: 8 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best
LBA Format 5 : Metadata Size: 64 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best
LBA Format 6 : Metadata Size: 128 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best

The drives defaulted to 512-byte LBA. Since these are enterprise SSDs and 4K was available, I reformatted both namespaces to format 3: 4096-byte sectors with no metadata.

This erases all data on the drive.

1
2


nvme format /dev/nvme0n1 -l 3
nvme format /dev/nvme1n1 -l 3

After formatting, nvme list should show 4 KiB + 0 B:

1
2
3


Node Model Namespace Usage Format FW Rev
/dev/nvme0n1 INTEL SSDPE2MX450G7 0x1 450.10 GB / 450.10 GB 4 KiB + 0 B MDV10290
/dev/nvme1n1 INTEL SSDPE2MX450G7 0x1 450.10 GB / 450.10 GB 4 KiB + 0 B MDV10290

Open the IPMI KVM

In the OVHcloud console, open the IPMI KVM.

The KVM launches through an old Java JNLP applet.

One annoying detail: the IP address you use to access the OVHcloud console should be the same public client IP used when opening the IPMI KVM. I hit a failure mode where the KVM was blocked because I accessed them through different egress IPs.

The machine uses an Intel server board. The useful hotkeys are:

F2: BIOS setup.
F6: one-time boot menu.

The boot logo confirms it is an Intel board:

You can mount a virtual ISO from the JViewer client:

Then use F6 and choose the virtual CD-ROM:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


Please select boot device:

UEFI IPv4: Intel I210 Network 00 at Baseboard
UEFI IPv4: Intel I210 Network 00 at Baseboard 2
UEFI IPv6: Intel I210 Network 00 at Baseboard
UEFI IPv6: Intel I210 Network 00 at Baseboard 2
Launch EFI Shell
Enter Setup
UEFI Virtual CDROM 1.00 # Choose this one!
UEFI Misc Device

Do Not Stream the Proxmox ISO Through JViewer

My first attempt was to mount the Proxmox installer ISO directly in JViewer and boot it.

That technically works, but it was unusably slow. Even when I ran JViewer from an OVH server (which should have enough bandwidth), virtual media throughput was capped at around 64 KB/s. Installing Proxmox by streaming a full ISO through that path would take forever.

The fix is to boot a tiny netboot image first.

Boot Proxmox Through netboot.xyz

Download the netboot.xyz UEFI ISO, mount that ISO in JViewer, and boot it. The image is tiny, so the slow virtual media path is no longer a problem.

In netboot.xyz, choose:

1
2
3


Linux Network Installs (64-bit)
Proxmox
Proxmox VE Text Installer

You can choose the debug installer if you want shell access between install stages. That is useful when something goes wrong.

netboot.xyz downloads the Proxmox installer over the server’s own network connection and then boots it. OVHcloud provides DHCP even for dedicated servers, so the installer had network access without manual IP configuration.

One caveat: after netboot.xyz hands off to the Proxmox installer, the installer does not use a serial console, so Serial over LAN will not work. You still need the JViewer window to see and control the installation.

Install Proxmox on ZFS RAID1

In the Proxmox installer, choose ZFS RAID1 across both NVMe drives.

I intentionally did not give the full SSDs to rpool. The installer has an hdsize option in advanced storage options. I set it to 128 GiB so the root pool would be a 2-way mirror, leaving the remaining space on both SSDs unused for a later scratch pool.

This is the important part: decide this during installation. Growing ZFS into extra space is easy. Shrinking an existing ZFS pool is not.

The final root layout on my machine:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


nvme0n1 419.2G disk
|-nvme0n1p1 1000K part
|-nvme0n1p2 1G part vfat
|-nvme0n1p3 128G part zfs_member
`-nvme0n1p4 290.2G part zfs_member

nvme1n1 419.2G disk
|-nvme1n1p1 1000K part
|-nvme1n1p2 1G part vfat
|-nvme1n1p3 128G part zfs_member
`-nvme1n1p4 290.2G part zfs_member

p3 on both drives is the mirrored root pool:

1
2
3
4
5
6
7
8
9


pool: rpool
state: ONLINE
config:

 NAME STATE
 rpool ONLINE
 mirror-0 ONLINE
 nvme-INTEL_SSDPE2MX450G7_CVPF71620037450RGN-part3 ONLINE
 nvme-INTEL_SSDPE2MX450G7_CVPF721600N5450RGN-part3 ONLINE

p4 on both drives is the later scratch pool.

Optional Partition Alignment Fix

The Proxmox installer did not give me a perfectly round 1 MiB-aligned final size for the ZFS partition (nvmeXn1p3). If you care about this, fix it immediately after installation while the layout is still simple.

The safe method is:

Use fdisk /dev/nvmeXn1
List partitions with p and note the start/end sector of part 3.
Delete d part 3
Create a new part 3 with the same start sector and a new end sector. Increase the end sector a bit, align end sector - 1 to 1M (256 4k sectors).
Do not wipe the ZFS signature.
Write changes with w.
Ask ZFS to expand into the recreated partition.

Example:

1

zpool online -e rpool <device>

Note that if you expanded the ZFS part too little (smaller than metaslabs size, typically 1GiB), ZFS may not actually expand the pool to use the empty space. This is fine.

Find the exact <device> with:

1

zpool status -v rpool

This is not required for a working system. I did it because I wanted the 128 GiB root partition to be exact and aligned. So later partition 4 will not be misaligned either.

Fix OVHcloud Boot-to-Disk

After a manual install, the server may not boot straight into Proxmox even though the installation succeeded.

OVHcloud bare-metal boot is not just “BIOS loads local disk”. The normal path is roughly:

The server PXE-boots from the public interface.
OVHcloud DHCP gives the server its public IP and an iPXE loader.
iPXE queries OVHcloud’s internal boot service.
The boot service returns a script based on your configured boot mode.
In boot-to-disk mode, iPXE uses sanboot with an EFI bootloader path.

If you installed through the OVHcloud panel, OVHcloud knows the EFI bootloader path. If you installed manually, that path can be missing or wrong.

When sanboot fails, iPXE falls back to rEFInd. rEFInd scans the EFI system partitions and may choose the wrong bootloader. In my case it picked memtest first:

1
2
3
4


rEFInd - Booting OS

Starting memtest86+x64.efi
Using load options ''

The rEFInd menu showed both memtest and systemd-boot:

1
2
3
4


Boot memtest86+x64.efi from 1021 MiB FAT volume
Boot EFI\systemd\systemd-bootx64.efi from 1021 MiB FAT volume
Boot memtest86+x64.efi from 1021 MiB FAT volume
Boot EFI\systemd\systemd-bootx64.efi from 1021 MiB FAT volume

For Proxmox installed with UEFI and root-on-ZFS, the bootloader is systemd-boot:

1

\EFI\systemd\systemd-bootx64.efi

Set that path through the OVHcloud CLI:

1
2


ovhcloud login
ovhcloud baremetal edit nsXXX.ip-X-X-X.eu --efi-bootloader-path '\EFI\systemd\systemd-bootx64.efi'

After that, boot-to-disk should go directly into Proxmox instead of falling through to rEFInd.

Basic Post-Install ZFS Settings

Enable TRIM on the root pool and set basic dataset properties:

1
2
3
4
5
6


zpool set autotrim=on rpool
zfs set atime=off rpool
# The default 'on' is also lz4. Just set it to lz4 explicitly to be sure.
zfs set compression=lz4 rpool
# 128k is the default (this is good for HDDs). But 16-32k is better for NVMe array and workloads with small files, which is the case for most VMs/CTs and Proxmox OS. I am using 2-way mirror so I don't need to worry about 4k ashift adding up.
zfs set recordsize=16k rpool

recordsize affects file datasets, not zvol block devices. For VM disks backed by ZFS zvols, check the Proxmox storage block size instead. On new Proxmox installations the default is already 16K; older installations may still use 8K. You can check or change it in:

1

Datacenter -> Storage -> local-zfs -> Edit -> Block Size

I also use a few ZFS module options:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# You may remove zfs_arc_max set by the installer if you want to maximize ZFS ARC. Note that ARC may not shrink fast enough under memory pressure, so keep this in mind.

cat >/etc/modprobe.d/zfs.conf <<'EOF'
options zfs zfs_txg_timeout=30
options zfs zfs_trim_txg_batch=128
options zfs zfs_dirty_data_sync_percent=80
options zfs zfs_delay_min_dirty_percent=95
EOF

update-initramfs -u -k all

The intent:

zfs_txg_timeout=30: reduce idle write frequency.
zfs_trim_txg_batch=128: make TRIM batching less tiny for NVMe.
zfs_dirty_data_sync_percent=80 and zfs_delay_min_dirty_percent=95: delay throttling until the dirty-data situation is actually serious.

These are not universal defaults. They fit this small host because the workload is mostly personal VMs/CTs and scratch tasks, not a database with strict latency guarantees.

Reduce Unnecessary Host Writes

I am not using Proxmox clustering on this server, so I disabled the cluster HA services:

1
2
3


systemctl disable --now pve-ha-crm.service
systemctl disable --now pve-ha-lrm.service
systemctl disable --now corosync.service

Limit persistent journald usage:

1
2


sed -i 's/.*SystemMaxUse.*/SystemMaxUse=128M/g' /etc/systemd/journald.conf
systemctl restart systemd-journald

Configure BBR

Use BBR with fq.

On Debian/Proxmox, a later default sysctl file can set net.core.default_qdisc=fq_codel after your file during boot. Use a late filename:

1
2
3
4
5
6


cat >/etc/sysctl.d/99-bbr.conf <<'EOF'
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
EOF

sysctl --system

Check it:

1

sysctl net.core.default_qdisc net.ipv4.tcp_congestion_control

Expected:

1
2


net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

Configure Swap

Always have swap when using ZFS, even with enough RAM. It gives the kernel somewhere to go if ARC does not shrink quickly enough under memory pressure, and it reduces the chance that the OOM killer targets useful applications.

I use ZRAM swap:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


git clone --depth=1 https://github.com/foundObjects/zram-swap.git
cd zram-swap
./install.sh
cd ..
rm -rf zram-swap

cat >/etc/sysctl.d/zram.conf <<'EOF'
vm.swappiness = 180
vm.watermark_boost_factor = 0
vm.watermark_scale_factor = 125
vm.page-cluster = 0
EOF

sysctl --system

You may use disk swap if you want.

My current host has a 38.8 GiB ZRAM swap device:

1
2


NAME TYPE SIZE USED PRIO
/dev/zram0 partition 38.8G 0B 15

SSH and Serial Console

I like SSH sessions to die quickly when the client disappears unexpectedly:

1
2


ClientAliveInterval 60
ClientAliveCountMax 3

Put that in sshd_config or a file under sshd_config.d, then restart SSH.

For emergency access, enable OVHcloud serial-over-LAN by adding a serial console to the Proxmox kernel command line:

1
2
3


additional_cmdline="console=tty0 console=ttyS0,115200n8"
sed -i "s|$| $additional_cmdline|" /etc/kernel/cmdline
proxmox-boot-tool refresh

On my host, /etc/kernel/cmdline now contains:

1

root=ZFS=rpool/ROOT/pve-1 boot=zfs vmlinuz video=vesafb:ywrap,mtrr initrd=initrd.magic console=tty0 console=ttyS0,115200n8

Scratch Storage

The root pool uses 128 GiB on each SSD. The remaining 290.2 GiB on each SSD is free for scratch storage.

I considered two options:

mdadm RAID0 + LVM thin + ext4/XFS.
ZFS striped pool.

The mdadm option may win in some microbenchmarks, but the management cost is higher. I used a ZFS striped pool because it is simple and good enough.

This pool has no redundancy. If either SSD fails, the scratch pool is gone. That is fine for my use case because it is scratch space.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


zpool create -f \
 -o ashift=12 \
 -o autotrim=on \
 scratchpool \
 /dev/disk/by-id/nvme-INTEL_SSDPE2MX450G7_CVPF721600N5450RGN-part4 \
 /dev/disk/by-id/nvme-INTEL_SSDPE2MX450G7_CVPF71620037450RGN-part4

zfs set recordsize=16k scratchpool
zfs set atime=off scratchpool
zfs set compression=lz4 scratchpool
zfs set logbias=throughput scratchpool

zfs create scratchpool/unsafe
zfs set sync=disabled scratchpool/unsafe
zfs set copies=1 scratchpool/unsafe

Current state:

1
2
3
4
5
6
7
8


pool: scratchpool
state: ONLINE
config:

 NAME STATE
 scratchpool ONLINE
 nvme-INTEL_SSDPE2MX450G7_CVPF721600N5450RGN-part4 ONLINE
 nvme-INTEL_SSDPE2MX450G7_CVPF71620037450RGN-part4 ONLINE

Properties:

1
2
3


NAME AVAIL MOUNTPOINT RECSIZE COMPRESS ATIME SYNC COPIES
scratchpool 562G /scratchpool 16K lz4 off standard 1
scratchpool/unsafe 562G /scratchpool/unsafe 16K lz4 off disabled 1

Use scratchpool/unsafe only for data you can recreate. sync=disabled lies to applications about sync writes.

mdadm + LVM Alternative

If you want the non-ZFS scratch layout, create one extra partition on each SSD and build RAID0:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


apt install -y mdadm

mdadm --create /dev/md0 \
 --verbose \
 --level=0 \
 --raid-devices=2 \
 --chunk=128K \
 /dev/disk/by-id/nvme-INTEL_SSDPE2MX450G7_CVPF721600N5450RGN-part4 \
 /dev/disk/by-id/nvme-INTEL_SSDPE2MX450G7_CVPF71620037450RGN-part4

mdadm --detail --scan >> /etc/mdadm/mdadm.conf
update-initramfs -u

For alignment:

mdadm chunk size: 128K. 128K is a good balance between performance and overhead for NVMe drives. If you want to maximize sequential performance, use larger values like 1M. But remember to align filesystem / LVM with this value.
Data disks: 2.
Stripe width: 256K.
LVM data alignment: 1M is fine because it is a multiple of 256K.

NVMe (physical) → mdadm RAID0 → LVM PV → LVM LV / thin pool → filesystem

When you create LVM on top of RAID-0, you must ensure:

LVM PE size = multiple of mdadm chunk size
LVM PV starts aligned to mdadm stripe boundary

Create the LVM layer:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# The default dataalignment value is 1M, which is a multiple of 256K, so it's fine.
pvcreate --dataalignment 1M /dev/md0
# Default VG extent size is often 4M, which is multiples of 1M data alignment, 
# which is also multiples of 256K stripe width. Using the default value is fine.
vgcreate --physicalextentsize 4M vg_scratch /dev/md0
# Thin pool chunk size defines how much physical space is reserved when any 
# part of a chunk is touched. It does not defined write size. So it can be 
# smaller than 256k. The default 64k is fine. I will just match it with the 
# stripe width = 256k.
# The default metadata size can be too big if you are using small driver 
# like 16G Optane drive. Calculate the desired metadata size (bytes)
# by `48 * (total size / chunk size)`.
lvcreate -l 100%FREE --thin vg_scratch/scratch_thin --chunksize 256k --poolmetadatasize 512M

When formatting a filesystem inside a VM or manually on the host, pass the stripe geometry:

1
2
3
4
5
6
7
8
9


# ext4
# stride = mdadm chunk size / filesystem block size = 128k / 4k = 32
# stripe-width = stride * disks = 32 * 2 = 64
mkfs.ext4 -b 4096 -E stride=32,stripe-width=64 /dev/vg_scratch/lv

# XFS
# su = mdadm chunk size = 128k
# sw = data disks = 2
mkfs.xfs -d su=128k,sw=2 /dev/vg_scratch/lv

Fast but unsafe mount options for scratch data:

1
2


ext4: noatime,nodiratime,nobarrier,data=writeback,commit=60
xfs: noatime,nodiratime,logbufs=8,logbsize=256k

For CTs, Proxmox does not make it convenient to set filesystem mount options on a normal CT mountpoint. If you need custom options, create the LV manually, mount it on the host, and bind-mount it into the CT:

1
2
3
4
5
6
7
8


lvcreate -V 16G -T vg_scratch/scratch_thin -n scratch_test
mkfs.xfs -d su=128k,sw=2 /dev/vg_scratch/scratch_test

mkdir -p /mnt/scratch_test
mount -o noatime,nodiratime,logbufs=8,logbsize=256k /dev/vg_scratch/scratch_test /mnt/scratch_test
chmod 777 /mnt/scratch_test

pct set 199 -mp0 /mnt/scratch_test,mp=/mnt/scratch_test

If your workload is random-write heavy and can fit on one SSD, skip RAID0 and LVM entirely. A single raw disk or single partition is often better for small random writes than a layered RAID0 + LVM-thin setup.

Using Scratch Storage in VMs and CTs

For VMs, add a virtual disk on the scratch pool. Enable discard and SSD emulation on the virtual disk. Inside the guest, enable periodic TRIM. When formatting the disk, make sure the filesystem is aligned to the underlying ZFS volsize or LVM stripe geometry. For example, if underlying ZFS volsize is 16K, use mkfs.ext4 -O bigalloc -C 16384 /dev/yourdisk. If the underlying storage is LVM+RAID, use the correct stride and stripe-width options when formatting.

For CTs, add a mountpoint on the scratch pool. ZFS mountpoints are automatically aligned. For LVM+RAID, make sure the filesystem is aligned to the underlying geometry. TRIM is handled by the host for ZFS mountpoints. For LVM+RAID, run pct fstrim on the CT.

To use the scratch pool for temporary data, you can use bind-mounts to mount common scratch directories to scratch disk mountpoints. This avoids changing the config for each applications such as Docker, containerd and etc.

Suppose the scratch disk is mounted at /mnt/unsafe, and you want to use it for Docker, containerd, and ~/.cache. Write the following lines in /etc/fstab:

1
2
3


/mnt/unsafe/docker /var/lib/docker none rbind 0 0
/mnt/unsafe/containerd /var/lib/containerd none rbind 0 0
/mnt/unsafe/.cache /root/.cache none rbind 0 0

1
2
3
4


mkdir -p /mnt/unsafe/{docker,containerd,.cache}
rm -rf /var/lib/docker /var/lib/containerd /root/.cache
mkdir -p /var/lib/docker /var/lib/containerd /root/.cache
mount -av

TRIM

For ZFS pools:

1
2


zpool set autotrim=on rpool
zpool set autotrim=on scratchpool

For VMs, enable discard and SSD emulation on virtual disks, then enable periodic TRIM inside the guest.

For CTs backed by ZFS mountpoints, the host handles it. For CTs backed by other storage such as LVM thin, run pct fstrim:

1
2
3
4
5
6
7
8
9


cat >/usr/local/bin/trimcts.sh <<'EOF'
#!/bin/bash
for id in $(pct list | awk '$2 == "running" {print $1}'); do
 echo "==> Trimming CT $id"
 pct fstrim "$id"
done
EOF

chmod +x /usr/local/bin/trimcts.sh

Then run it from cron or a systemd timer. For example, weekly from cron:

1

0 2 * * 7 /usr/local/bin/trimcts.sh >/var/log/trimcts.log 2>&1

Proxmox Networking on a Single Public IP

Proxmox creates a bridge by default and attaches the physical NIC to it. That is usually the right setup when the server is in your own LAN.

It is not right for this Kimsufi server.

The server has one usable public IPv4 address and one /128 IPv6 address. It does not have a routed public subnet for VMs and CTs. If VMs are bridged directly to the public interface, they do not get usable public addresses.

So I do this instead:

Put the public IPv4 and public IPv6 directly on eno1, not vmbr0. By default, Proxmox puts the public IP on the vmbr0 bridge, which is not what we want. You should move it to the physical interface.
Create vmbr0 as an internal bridge with no physical ports.
Put VMs and CTs on vmbr0.
NAT outbound traffic from vmbr0 to eno1.
Port-forward selected inbound ports to service CTs.

Current /etc/network/interfaces:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
 address <public-ipv4>/24
 gateway <public-ipv4-gateway>

iface eno1 inet6 static
 address <public-ipv6>/128
 gateway <public-ipv6-gateway>

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
 address 10.187.54.1/24
 bridge-ports none
 bridge-stp off
 bridge-fd 0

iface vmbr0 inet6 static
 address fc10:187:54::1/64

source /etc/network/interfaces.d/*

Enable forwarding:

1
2
3
4
5
6


cat >/etc/sysctl.d/99-forward.conf <<'EOF'
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF

sysctl --system

About NAT66: it is usually the wrong IPv6 design. If you have a routed IPv6 prefix, route it to your VM bridge. On this server I only have one /128, so NAT66 is the practical workaround if VMs/CTs need outbound IPv6.

NAT and Port Forwarding

NAT rewrites packet addresses as traffic crosses the host.

For outbound IPv4 NAT, a CT sends a packet like:

1

src=10.187.54.100:51514 dst=1.1.1.1:443

After MASQUERADE on the host:

1

src=<public-ipv4>:51514 dst=1.1.1.1:443

Linux conntrack records that translation so the reply can be translated back and forwarded to the CT.

There is a Proxmox-specific caveat when the firewall is enabled on a VM or CT network interface (firewall=1). Proxmox places that interface behind an fwbr* firewall bridge. With bridge netfilter enabled, conntrack can record the connection while it crosses the firewall bridge, before it reaches the host’s normal NAT path. The later FORWARD rule can accept the packet, but POSTROUTING may reuse the already tracked non-NATed connection instead of attaching MASQUERADE or SNAT. The packet then leaves the public interface with its private source address and never receives a reply.

Put the firewall-bridge tracking in a separate conntrack zone by adding these hooks to the vmbr0 stanza in /etc/network/interfaces:

1
2
3
4


 post-up iptables -t raw -C PREROUTING -i fwbr+ -j CT --zone 1 2>/dev/null || iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1 2>/dev/null || true
 post-up ip6tables -t raw -C PREROUTING -i fwbr+ -j CT --zone 1 2>/dev/null || ip6tables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
 post-down ip6tables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1 2>/dev/null || true

The -C/|| guard matters on a dual-stack vmbr0: ifupdown2 can run the hook once for each address family.

The IPv4 rules cover MASQUERADE/SNAT, while the IPv6 rules are also needed when using NAT66. A characteristic symptom is that NAT works for a guest without the Proxmox interface firewall but fails for an otherwise identical firewall-enabled guest. In conntrack output, the broken flow’s reply destination remains the guest’s private address instead of the host’s public address. Allowing all OUT and FORWARD traffic does not fix it because this is a conntrack/NAT-path problem, not a filter-policy rejection.

Inbound port forwarding is DNAT. A client connects to the public server:

1

src=198.51.100.20:41000 dst=<public-ipv4>:443

The host rewrites the destination before routing:

1

src=198.51.100.20:41000 dst=10.187.54.3:443

I use my own small tool, iptfwd, to manage these rules from a config file: charlie0129/iptfwd.

Example config for this topology:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


defaults:
 public_iface: eno1
 private_iface: vmbr0
 manage_filter: true
 enable_ip_forwarding: true

nat:
 - name: intranet-v4
 source: 10.187.54.0/24
 type: masquerade

 - name: intranet-v6
 source: fc10:187:54::/64
 type: snat
 to_source: <public-ipv6>

rules:
 - name: service-http-v4
 proto: tcp
 public_port: 80
 target: 10.187.54.3
 target_port: 80

 - name: service-https-v4
 proto: tcp
 public_port: 443
 target: 10.187.54.3
 target_port: 443

 - name: service-http-v6
 proto: tcp
 public_port: 80
 target: fc10:187:54::3
 target_port: 80

 - name: service-https-v6
 proto: tcp
 public_port: 443
 target: fc10:187:54::3
 target_port: 443

For IPv4, MASQUERADE is convenient. For NAT66, I prefer explicit SNAT to the public IPv6 address. IPv6 interfaces can have multiple addresses, and explicit SNAT makes the translated source predictable.

If you do not want to use iptfwd, the equivalent basic iptables rules are:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


iptables -t nat -A POSTROUTING -s 10.187.54.0/24 -o eno1 -j MASQUERADE
iptables -A FORWARD -i vmbr0 -o eno1 -j ACCEPT
iptables -A FORWARD -i eno1 -o vmbr0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

public_ipv6=<YOUR_PUBLIC_IPV6>
ip6tables -t nat -A POSTROUTING -s fc10:187:54::/64 -o eno1 -j SNAT --to-source "$public_ipv6"
ip6tables -A FORWARD -i vmbr0 -o eno1 -j ACCEPT
ip6tables -A FORWARD -i eno1 -o vmbr0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

apt install iptables-persistent
netfilter-persistent save

Before setting up DHCP, test NAT manually:

Create a CT on vmbr0.
Set IPv4 to something like 10.187.54.100/24.
Set IPv4 gateway to 10.187.54.1.
Set IPv6 to something like fc10:187:54::100/64.
Set IPv6 gateway to fc10:187:54::1.
Test:

1
2


ping www.google.com
ping6 www.google.com

If that fails, debug NAT before adding DHCP.

DHCP and DNS with Pi-hole

I use Pi-hole as DHCP and DNS for the private bridge because it has a nice web UI and makes static leases easy.

Create a small Alpine CT:

Bridge: vmbr0.
IPv4: 10.187.54.2/24.
IPv4 gateway: 10.187.54.1.
IPv6: fc10:187:54::2/64.
IPv6 gateway: fc10:187:54::1.
CPU: 1 core is enough.
RAM: 128 MB is enough.
Disk: 512 MB is enough.

Install Pi-hole:

1
2
3
4


apk update
apk add curl bash
curl -sSL https://install.pi-hole.net | bash
# You may disable query logs to reduce writes to the database.

If you forget the web password:

1

pihole setpassword

Reduce query-log database writes:

1

pihole-FTL --config database.DBinterval 1800

In the Pi-hole UI:

Enable DHCP for 10.187.54.0/24.
Set the router/gateway to 10.187.54.1.
Configure DNS as desired.
Disable NTP sync under Settings -> All Settings -> Network Time Sync. Uncheck ntp.ipv4/ipv6/sync.active. An unprivileged CT cannot set system time anyway.

Alpine log cleanup I use for small CTs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


cat >/etc/periodic/hourly/logtruncate <<'EOF'
#!/bin/sh

LOG_DIR="/var/log"
MAX_SIZE=$((4 * 1024 * 1024))

find "$LOG_DIR" -type f ! -name '*.0' | while read -r file; do
 size=$(stat -c %s "$file" 2>/dev/null || stat -f %z "$file" 2>/dev/null || echo 0)

 if [ "$size" -gt "$MAX_SIZE" ]; then
 logger -t logrotate "Rotating $file ($size bytes)"
 cp "$file" "$file.0"
 truncate -s 0 "$file"
 logger -t logrotate "Rotated $file"
 fi
done
EOF

chmod +x /etc/periodic/hourly/logtruncate
sed -i 's/^SYSLOGD_OPTS=.*/SYSLOGD_OPTS="-t -s 0"/' /etc/conf.d/syslog

cat >/etc/periodic/daily/apkcacheclean <<'EOF'
#!/bin/sh
find /var/cache/apk -type f -name '*.apk' -mtime +7 -delete
find /var/cache/apk -type f -name '*.tar.gz' -mtime +90 -delete
EOF

chmod +x /etc/periodic/daily/apkcacheclean

Host Monitoring

I keep monitoring tools in Podman containers instead of installing everything directly on the Proxmox host.

Install Podman:

1
2
3
4
5
6


# So Intel PCM can work.
echo msr >> /etc/modules
modprobe msr

# Install podman to run monitoring tools in containers. Do not include any unnecessary packages to keep the host clean and minimal. Remember to install aardvark-dns to make container DNS work.
apt install --no-install-recommends podman aardvark-dns

On current Debian/Proxmox, Podman should use overlay storage on ZFS:

1

podman info --format '{{.Store.GraphDriverName}}'

Expected:

1

overlay

Make podman-restart.service also handle containers with restart-policy=unless-stopped:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


mkdir -p /etc/systemd/system/podman-restart.service.d

cat >/etc/systemd/system/podman-restart.service.d/override.conf <<'EOF'
[Service]
ExecStart=/usr/bin/podman $LOGGING start --all --filter restart-policy=always --filter restart-policy=unless-stopped
ExecStop=/usr/bin/podman $LOGGING stop --all --filter restart-policy=always --filter restart-policy=unless-stopped
EOF

systemctl daemon-reload
systemctl enable podman-restart.service

Enable the Podman socket and Docker-compatible CLI:

1
2
3
4
5


systemctl enable --now podman.socket

apt install --no-install-recommends docker-cli docker-compose
docker context create podman --docker "host=unix:///run/podman/podman.sock"
docker context use podman

An example monitoring compose file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85


services:
 vmagent:
 container_name: vmagent
 image: victoriametrics/vmagent:v1.144.0
 restart: unless-stopped
 mem_limit: 512M
 environment:
 GOMEMLIMIT: 900MiB
 command:
 - -httpListenAddr=0.0.0.0:8428
 - -cacheExpireDuration=5m
 - -promscrape.config=/etc/vmagent/scrape.yml
 - -remoteWrite.url=https://XXX # Your VictoriaMetrics remote write endpoint
 - -remoteWrite.basicAuth.username=XXX
 - -remoteWrite.basicAuth.password=XXX
 - -remoteWrite.maxDiskUsagePerURL=128MiB
 - -remoteWrite.tmpDataPath=/var/vmagent
 - -remoteWrite.vmProtoCompressLevel=3
 - -remoteWrite.flushInterval=66s
 - -remoteWrite.label=instance=XXX
 extra_hosts:
 - "host.docker.internal:host-gateway"
 volumes:
 - ./vmagent/config:/etc/vmagent
 - ./vmagent/data:/var/vmagent

 smartctlexporter:
 container_name: smartctlexporter
 image: prometheuscommunity/smartctl-exporter:v0.14.0
 restart: unless-stopped
 mem_limit: 64M
 user: 0:0
 environment:
 GOMEMLIMIT: 110MiB
 privileged: true

 nodeexporter:
 container_name: nodeexporter
 image: prom/node-exporter:v1.11.1
 restart: unless-stopped
 mem_limit: 64M
 environment:
 GOMEMLIMIT: 110MiB
 privileged: true
 command:
 - --path.rootfs=/host
 - --web.listen-address=:9100
 network_mode: host
 pid: host
 volumes:
 - "/:/host:ro,rslave"

 podmanexporter:
 container_name: podmanexporter
 image: quay.io/navidys/prometheus-podman-exporter:v1.21.0
 mem_limit: 64M
 user: 0:0
 privileged: true
 environment:
 GOMEMLIMIT: 110MiB
 CONTAINER_HOST: unix:///run/podman/podman.sock
 restart: unless-stopped
 volumes:
 - /run/podman/podman.sock:/run/podman/podman.sock
 command:
 - --web.listen-address=:9882

 cgroupexporter:
 container_name: cgroupexporter
 image: ghcr.io/arianvp/cgroup-exporter:0.3.3-amd64
 mem_limit: 64M
 environment:
 GOMEMLIMIT: 110MiB
 restart: unless-stopped
 volumes:
 - /sys/fs/cgroup:/sys/fs/cgroup:ro
 command:
 - -listen-address=:3232

 pcm:
 container_name: pcm
 image: opcm/pcm
 mem_limit: 96M
 restart: unless-stopped
 privileged: true

Service CT

I keep public-facing services in a separate CT and forward only selected ports from the host.

Create an unprivileged Alpine CT on vmbr0, then:

1
2
3


apk add openssh openssh-server
rc-update add sshd default
rc-service sshd start

Add this to /etc/ssh/sshd_config:

1
2
3


PermitRootLogin yes
ClientAliveInterval 60
ClientAliveCountMax 3

Install Podman inside the CT:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


cat >/usr/local/bin/enablecgroup2nesting <<'EOF'
#!/bin/sh
if [ -f /sys/fs/cgroup/cgroup.controllers ]; then
 echo "Enabling cgroup v2 nesting"
 mkdir -p /sys/fs/cgroup/init
 xargs -rn1 < /sys/fs/cgroup/cgroup.procs > /sys/fs/cgroup/init/cgroup.procs || :
 sed -e 's/ / +/g' -e 's/^/+/' < /sys/fs/cgroup/cgroup.controllers \
 > /sys/fs/cgroup/cgroup.subtree_control
fi
EOF
chmod +x /usr/local/bin/enablecgroup2nesting

cat >/etc/init.d/cgroup2nesting <<'EOF'
#!/sbin/openrc-run

description="Enable nesting of cgroup2."

depend()
{
 keyword -docker -podman -prefix -systemd-nspawn -vserver -wsl
 after sysfs
}

start()
{
 ebegin "Enabling cgroup v2 nesting"
 /usr/local/bin/enablecgroup2nesting
 eend $?
}
EOF
chmod +x /etc/init.d/cgroup2nesting
rc-update add cgroup2nesting default
rc-service cgroup2nesting start

apk add podman iptables
sed -i 's/^#log_size_max =.*/log_size_max = 1048576/' /etc/containers/containers.conf

rc-update add podman
rc-service podman start

apk add docker-cli docker-cli-compose
docker context create podman --docker "host=unix:///run/podman/podman.sock"
docker context use podman

Also check out the Pi-Hole setup section for Alpine log cleanup and other small CT optimizations. You must do this because Alpine does not have log rotation by default. You disk will fill up if you do not manage logs.

Then run Caddy or Traefik in that CT for reverse proxy and TLS termination. Forward 80 and 443 from the host to the service CT with iptfwd or iptables.

If you use a privileged CT with Podman over ZFS, you may need an overlay mount helper:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


cat >/usr/local/bin/zfsoverlaymount <<'EOF'
#!/bin/sh
/bin/mount -t overlay overlay "$@"
EOF
chmod +x /usr/local/bin/zfsoverlaymount

cat >/etc/containers/storage.conf <<'EOF'
[storage.options]
mount_program = "/usr/local/bin/zfsoverlaymount"
EOF

I prefer unprivileged CTs unless there is a specific reason not to use them.

SMTP Notifications

By default, Proxmox may send mail directly and anonymously to the email address you entered during installation. Some providers reject that mail.

Configure a real SMTP target instead:

1

Datacenter -> Notifications -> Notification Targets -> Add -> SMTP

Final State

The host ended up with:

Proxmox VE 9.2.
Mirrored ZFS root pool on 128 GiB from each NVMe SSD.
Striped ZFS scratch pool on the remaining SSD space.
4K LBA on both NVMe drives.
Private-only vmbr0 for VMs and CTs.
IPv4 NAT and IPv6 NAT66 from vmbr0 to eno1.
80/443 forwarded to a service CT.
Pi-hole handling private DHCP and DNS.
Podman-based host monitoring.

For 19.90 USD/month, this is a surprisingly useful little virtualization box.

Minimal Alpine Linux on a 1 GB Btrfs Root Disk

Thu, 25 Jun 2026 15:00:00 +0800

I sometimes need a tiny cloud instance that only forwards traffic. CPU and memory can be small, and the disk is mostly wasted space. On one Alibaba Cloud ecs.t6-c4m1.large instance, a 1 GB root disk on a long contract can push the cost down to about $0.40/month, but a normal Linux install leaves too little usable space.

The trick is not complicated:

Use Alpine Linux, because the base system is small.
Keep /boot tiny. A 512 MB boot partition is absurd on a 1 GB disk.
Put / on Btrfs with transparent compression. Application files, package metadata, and logs usually compress well.
Do not allocate disk swap. Use zram later if the machine needs swap-like behavior.

This post is the install recipe I use to build a minimal BIOS/MBR Alpine image, then optionally convert it to QCOW2 for reuse.

If you would rather not do this by hand, I later automated the whole thing: Building Alpine Linux Disk Images Without a VM is a script that builds the same class of image as a file, with no VM and no interactive steps, and boots under both BIOS and UEFI. This post is still the explanation of why the pieces are the way they are.

Assumptions

This guide targets a very specific VM shape:

Legacy BIOS boot with MBR and Syslinux/Extlinux.
One small ext4 /boot partition and one Btrfs root partition.
No disk encryption, no LVM, no UEFI, no separate /var.
The whole disk can be destroyed.

If your VM boots with UEFI, use an EFI system partition and GRUB instead. If your disk is NVMe, adapt the partition names to /dev/nvme0n1p1 and /dev/nvme0n1p2.

The final layout is:

Partition	Size	Filesystem	Mountpoint
`/dev/vda1`	64 MB	ext4	`/boot`
`/dev/vda2`	Rest of disk (~500MB is fine)	Btrfs	`/`

64 MB is enough for this single-kernel Alpine image. The boot partition usage should be around 30M with Alpine 3.23 virt kernel (6.18). If you plan to keep multiple kernels or use a larger bootloader setup, use 128 MB instead.

Before you start installing, create a VM with a ~512MB disk (yes, really). CPU and RAM can be small, because the install process is not resource-intensive. After the install, you can resize the disk to 1 GB or larger.

Always go with a small disk first because the disk image can be expanded easily, but shrinking a disk is not trivial.

Partition The Disk

Set the disk variables so the commands below are harder to mistype:

1
2
3


DISK=/dev/vda
BOOT=/dev/vda1
ROOT=/dev/vda2

Now create a DOS/MBR partition table with a tiny boot partition:

1

fdisk "$DISK"

Inside fdisk, enter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


o # new empty DOS partition table
n # new partition
p # primary
1 # partition number
2048 # first sector, 1 MiB aligned
+64M # size
a # toggle bootable flag
1

n # new partition
p # primary
2 # partition number
133120 # first sector immediately after the 64 MB partition
 # press Enter for the default end sector

p # verify the table
w # write changes

The important details are:

/dev/vda1 should be bootable.
/dev/vda1 should start at sector 2048.
/dev/vda2 should start at sector 133120.
The second partition should use the rest of the disk.

Check that the kernel sees the new partitions:

1

ls -l /dev/vda*

If /dev/vda1 and /dev/vda2 do not appear, reboot the live ISO. Alpine’s manual disk setup docs also recommend rebooting after manual partition creation when needed.¹

Start Alpine Setup

Download the latest Alpine ISO from https://alpinelinux.org/downloads/. I usually use the x86_64 architecture and the virtual variant. As this will be used in virtualized environments, the virt kernel is smaller.

Boot the Alpine ISO and log in as root. The live environment has no root password by default.

Run the normal installer first:²

1

setup-alpine

Answer the usual questions for keyboard, hostname, network, DNS, root password, timezone, mirror, SSH, and NTP. I usually choose chrony for NTP.

When it asks which disk to use, type:

1

none

This keeps the base configuration but skips automatic partitioning and formatting. For the later diskless-mode prompts, also choose none for configs. You can keep the default apk cache dir (/var/cache/apk).

Format And Mount

Install the filesystem tools in the live environment:

1

apk add btrfs-progs e2fsprogs

Format /boot as ext4 and / as Btrfs:

1
2


mkfs.ext4 -m 0 -L boot "$BOOT"
mkfs.btrfs -f -L alpine-root "$ROOT"

Mount root with compression enabled:

1
2
3
4
5
6


# adjust the compression level as desired. zstd:1 is fast and has reasonable compression. zstd:3 is slower but compresses better. Any higher level is usually overkill and comes with diminishing returns.
mount -t btrfs -o rw,relatime,compress=zstd:3,ssd,discard=async,space_cache=v2 "$ROOT" /mnt
btrfs property set /mnt compression zstd:3

mkdir -p /mnt/boot
mount -t ext4 "$BOOT" /mnt/boot

I keep /boot on ext4 because it is small and predictable with this Syslinux/MBR setup. On a tiny image, boring boot is good boot.

Install Alpine

Install Alpine into the mounted filesystem:

1

setup-disk -v -m sys -s 0 /mnt

The options mean:

-m sys: install a traditional persistent system to disk.
-s 0: do not create disk swap.
/mnt: use the partitions already mounted under /mnt.

This is the standard setup-disk -m sys /mnt flow, just with a manually prepared root filesystem.³

Verify fstab And initramfs

Check /mnt/etc/fstab before rebooting:

1

vi /mnt/etc/fstab

The root entry should include Btrfs compression. Mine looks like this:

1
2


UUID=... / btrfs rw,relatime,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/ 0 1
UUID=... /boot ext4 rw,relatime 0 2

Remove any stale cdrom or usbdisk entries if the installer generated them.

Also check that the installed initramfs knows about Btrfs:

1

grep -w btrfs /mnt/etc/mkinitfs/mkinitfs.conf

If btrfs is missing from features="...", add it before rebooting and regenerate the initramfs from a chroot. Alpine explicitly calls this out for manual Btrfs root installs.⁴

Fix The MBR Bootloader

On these tiny MBR images, I have seen the first-stage bootloader not get written correctly. Reinstalling the Syslinux MBR is cheap insurance:

1
2


apk add syslinux
dd bs=440 count=1 conv=notrunc if=/usr/share/syslinux/mbr.bin of="$DISK"

Use gptmbr.bin instead only if you are using GPT. For the DOS partition table above, mbr.bin is the right file.⁵

Now unmount and reboot:

1
2
3
4
5


cd /
sync
umount /mnt/boot
umount /mnt
reboot

After the first boot, log in and check the actual space usage:

1
2


df -h /
btrfs filesystem usage /

Convert The Disk To QCOW2

Once the image is configured the way you want, shut it down and convert the raw disk from the host:

1

qemu-img convert -f raw -c -p -O qcow2 /dev/mapper/ex950-vm--300--disk--0 alpine-minimal.qcow2

Replace the source device with your VM disk. Do this from a stopped VM or a consistent snapshot, not from a running writable system.

You can now reuse the QCOW2 as a base image for other VMs or cloud instances.

Restore From The Provider’s Original OS

Some cloud providers do not let you upload or boot a custom disk image. If you still have VNC/serial-console access, another way is to boot the provider’s original Linux image, download your Alpine QCOW2 there, convert it to raw, then reboot into the original OS initramfs and overwrite the whole disk from there.

The important reason for the two-stage flow is tooling: the original OS probably has curl and qemu-img, while the initramfs usually does not. The initramfs is only used for the final dd, because at that point the real root filesystem is not mounted and can be safely overwritten.

This is destructive. Double-check the target disk and the old root partition before running dd.

First, in the provider’s original OS. This filesystem needs enough free space to hold both the downloaded QCOW2 and the converted raw image:

1
2
3
4
5


# Download the Alpine minimal image in qcow2 format and convert it to raw.
# Do this in the original OS because the initramfs probably does not have
# curl or qemu-img.
curl -L "<qcow2 link>" -o /os.qcow2
qemu-img convert -f qcow2 -O raw /os.qcow2 /os.raw

Then use the provider console or VNC to reboot into initramfs. On a GRUB-based original OS, select the normal boot entry, press e, find the Linux kernel command line, append:

1

break=premount

Then press F10 or Ctrl-x to boot. This should drop you into an initramfs shell before the root filesystem is mounted.

From the initramfs shell:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


# Find the old root partition and the target disk.
# Example:
# old root partition: /dev/sda2
# target disk: /dev/sda
cat /proc/partitions

mkdir /tmp/rootfs
cd /tmp

# Mount the old root filesystem read-only to copy the prepared image.
# Replace /dev/sda2 with the root partition from the original OS, and
# replace ext4 if the provider image uses another filesystem.
mount -t ext4 -o ro /dev/sda2 rootfs

# Copy the raw image to initramfs memory so the old root can be unmounted
# before overwriting the disk.
#
# This requires enough RAM for /os.raw. If the VM is too small, attach a
# temporary rescue disk or use a rescue system that can stream the image
# from the network instead.
dd if=rootfs/os.raw of=os.raw

umount rootfs

# This overwrites the whole disk. Replace /dev/sda with the target disk.
dd if=os.raw of=/dev/sda

reboot

After booting into Alpine, apply provider-specific network settings. For a static IPv4 setup:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


cat <<'EOF' > /etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
 address X.X.X.X/24
 gateway X.X.X.X
EOF

cat <<'EOF' > /etc/resolv.conf
nameserver 1.1.1.1
nameserver 1.0.0.1
EOF

setup-hostname XXXX

rc-service networking restart

If the provider uses DHCP, keep the interface as DHCP instead and let Alpine request the address normally.

Restore Without A Console Or Rescue Mode

The cheapest tiers take the rest of it away too: no custom-image import, no VNC, no serial console, no rescue mode. The stock OS and SSH are all you get. The initramfs trick above is unreachable there, because its one console step — interrupting GRUB — has no SSH equivalent: an initramfs shell has no network stack and no sshd.

What still works is having a script arrange that environment for you. bin456789/reinstall rewrites the stock bootloader to boot a minimal Alpine that runs entirely in RAM, and from there downloads your image and dds it over the whole disk before rebooting into it. No console is involved at any point. You watch the progress over SSH, and if the download or the write fails, the RAM system keeps its sshd up, so a failure is a retry by hand rather than a reinstall ticket.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# From the provider's stock OS, as root. This is destructive: it will
# wipe the whole disk, other partitions included.
#
# Mainland-China boxes often cannot reach raw.githubusercontent.com; the
# project mirrors the script at cnb.cool for exactly that case.
curl -O https://raw.githubusercontent.com/bin456789/reinstall/main/reinstall.sh
# curl -O https://cnb.cool/bin456789/reinstall/-/git/raw/main/reinstall.sh

# Prompts for a username/password, used to log in over SSH and watch.
# .gz / .xz / .zst compressed raw images are accepted directly.
bash reinstall.sh dd --img="https://example.com/alpine.img.gz"

# Still reversible until here: `bash reinstall.sh reset` restores the
# original bootloader config.
reboot

Before it does anything, the script asks for a username and a password, and it is worth being clear about what they are not: the new system’s login. They exist only inside the RAM installer environment — they are what you would type to SSH back in while the dd runs, to watch progress or clean up after a failure. A Linux image gets nothing injected; the script’s own end-of-run summary says as much, printing Password: [Depends on image] under “After Install”. So press Enter twice — root, then a random password the summary displays for copying — and let whatever the image baked in be the machine’s real credentials. --ssh-key replaces that password with one of your public keys for the installer environment, if typing nothing at all is the goal.

When you do log in mid-install, expect a host-key mismatch: the RAM environment presents its own host keys, and so does the finished image, so the client will complain about the stock OS’s key on the way in and about both of them after the final reboot. That is the process working, not something broken — clear the old entry with ssh-keygen -R or accept the new key each time. On a NAT’d box the installer’s sshd still listens on port 22 internally, so the provider’s forwarded port keeps working throughout.

Four things to get right, all of them consequences of there being no console:

The image URL must be reachable from the VPS. The download runs inside the RAM environment, over the same network the stock OS uses. For a box in mainland China that usually rules out GitHub-hosted files; one of my own servers or local object storage works better.

The image has to work as-is. dd mode does not modify a Linux image, so the network settings, the SSH keys, and the sshd port must be baked into the image before flashing — the “apply provider-specific network settings” step above has to happen inside the image, not on the machine afterwards. On a NAT’d box, where the provider forwards one public port to (say) internal port 22, changing the sshd port in the image orphans that forward and the machine. If the image comes from the builder, the 10-network and 20-ssh hooks are where this goes.

Check whether the stock OS keeps its root on LVM before rebooting. The script stages reinstall-vmlinuz and reinstall-initrd in the stock OS’s root directory and lets GRUB find them with search --file. GRUB’s LVM support is incomplete — thin-provisioned volumes are unsupported outright, and some perfectly ordinary LVM roots also defeat it for reasons that are not understood (issue #355). When that happens, GRUB stops at file '/reinstall-vmlinuz' not found and the RAM environment is never reached. On a machine with a console that is a visible error; on this kind of machine the box reboots and simply never speaks again. One lsblk settles it beforehand — if / sits on an lvm device, copy both files to /boot, which Debian-style LVM installs keep on a plain partition that GRUB reads reliably:

1
2


lsblk -o NAME,TYPE,FSTYPE,MOUNTPOINTS # is / on an lvm device?
cp /reinstall-vmlinuz /reinstall-initrd /boot/

The copies are harmless if they were not needed: search --file scans every filesystem GRUB can read, so it finds the /boot copies whenever the LVM ones are invisible. The workaround is the tool author’s own, from the issue thread. If /boot is only a directory inside the LVM root, there is no plain partition to stage onto, and I would not run this flow on such a box without a console.

Audit before the final reboot. --hold 2 stops after the dd finishes but before the reboot. SSH back in, mount the freshly written root and boot partitions read-write, and check /etc/network/interfaces, the authorized keys, and the bootloader config while a mistake is still fixable with a text editor. It is the last moment anything is.

The RAM environment is small — the project lists 256 MB of RAM as the minimum — so this works on boxes too tiny for anything else. To drive the final dd by hand instead, bash reinstall.sh alpine --hold 1 boots the same RAM Alpine and stops there, nothing written; it is also a cheap dry run for whether that environment’s networking works on the provider’s NAT at all. And if the machine still does not come back after the last reboot, the floor is the provider’s own reinstall button, back to the stock OS — annoying, not fatal.

Things I Bake Into The Image

The base install above is intentionally small. The sections below are optional knobs I usually apply before making the reusable image.

SSH

For a private forwarding box, I allow root SSH and TCP forwarding. Use key auth and firewall rules if this machine is reachable from the public Internet.

Add or change these lines in /etc/ssh/sshd_config:

1
2
3
4


PermitRootLogin yes
AllowTcpForwarding yes
ClientAliveInterval 60
ClientAliveCountMax 3

Restart SSH:

1

rc-service sshd restart

ClientAliveInterval 60 plus ClientAliveCountMax 3 makes dead SSH forwarding sessions disappear after roughly three minutes instead of waiting for long TCP timeouts.

Community Repository

Many useful packages live in community.

Either run:

1

setup-apkrepos -c

Or edit /etc/apk/repositories and uncomment the matching community repository:

1

# http://dl-cdn.alpinelinux.org/alpine/<alpine version>/community

Then refresh indexes:

1

apk update

IPv6 DHCP

SLAAC may work without any configuration. Some cloud providers need DHCPv6, in which case add an IPv6 stanza to /etc/network/interfaces:

1
2
3


auto eth0
iface eth0 inet dhcp
iface eth0 inet6 dhcp

Install the DHCP and interface tooling if your image does not already have it:

1
2


apk add dhcpcd ifupdown-ng
rc-service networking restart

Chrony

If NTP was not configured during install:

1
2
3
4
5
6
7
8
9


apk add chrony
vi /etc/chrony/chrony.conf

# Use `ntp.aliyun.com` if you are in mainland China; `pool.ntp.org`
# is often unreliable from there. Elsewhere, the default pool is fine.
#
# Also add `makestep 1.0 -1` so chrony can correct large time offsets
# on startup, which is common in VMs. You can remove the default
# `initstepslew ...` line if it exists.

The result should look like this:

1
2
3
4
5
6


pool pool.ntp.org iburst
initstepslew 10 pool.ntp.org
driftfile /var/lib/chrony/chrony.drift
rtcsync
cmdport 0
makestep 1.0 -1

Enable it:

1
2


rc-update add chronyd default
rc-service chronyd restart

Log Caps

Alpine does not install a full log rotation stack by default. On a 1 GB disk, I prefer a tiny periodic script that keeps one backup and truncates logs in place so daemons can continue writing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


cat <<'EOF' > /etc/periodic/hourly/logtruncate
#!/bin/sh

# Rotate log files larger than 4M.
# Keep one .0 backup and truncate the original in place.

LOG_DIR="/var/log"
MAX_SIZE=$((4 * 1024 * 1024)) # 4M in bytes

find "$LOG_DIR" -type f ! -name '*.0' | while read -r file; do
 # Works with both GNU and BusyBox stat.
 size=$(stat -c %s "$file" 2>/dev/null || stat -f %z "$file" 2>/dev/null || echo 0)

 if [ "$size" -gt "$MAX_SIZE" ]; then
 logger -t logtruncate "Rotating $file ($size bytes)"

 # Copy current content to .0 backup, overwriting the old backup.
 cp "$file" "$file.0"

 # Preserve the inode so daemons can continue writing.
 truncate -s 0 "$file"

 logger -t logtruncate "Rotated $file (backup: $file.0)"
 fi
done
EOF

chmod +x /etc/periodic/hourly/logtruncate
rc-update add crond default
rc-service crond start

Then tell BusyBox syslog not to rotate by itself:

1
2


sed -i 's/^SYSLOGD_OPTS=.*/SYSLOGD_OPTS="-t -s 0"/' /etc/conf.d/syslog
rc-service syslog restart

APK Cache Cleanup

If apk caching is enabled later, clean old packages periodically:

1
2
3
4
5
6
7


cat <<'EOF' > /etc/periodic/daily/apkcacheclean
#!/bin/sh
find /var/cache/apk -type f -name '*.apk' -mtime +7 -delete
find /var/cache/apk -type f -name '*.tar.gz' -mtime +90 -delete
EOF

chmod +x /etc/periodic/daily/apkcacheclean

Timezone

1
2


apk add tzdata
setup-timezone -z Asia/Shanghai # replace with your timezone

ZeroTier

Alpine’s ZeroTier package can lag behind upstream, so I usually use static binaries from zerotier-static.

After placing zerotier-one at /usr/local/bin/zerotier-one, create an OpenRC service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


cat <<'EOF' > /etc/init.d/zerotier-one
#!/sbin/openrc-run

depend() {
 after network-online
 want cgroups
}

start_pre() {
 /sbin/modprobe tun
}

supervisor=supervise-daemon
name=zerotier-one
command="/usr/local/bin/zerotier-one"
command_args=" \
 >>/var/log/zerotier-one.log 2>&1"

output_log=/var/log/zerotier-one.log
error_log=/var/log/zerotier-one.log

pidfile="/var/run/zerotier-one.pid"
respawn_delay=5
respawn_max=0
EOF

chmod +x /etc/init.d/zerotier-one
rc-update add zerotier-one default
rc-service zerotier-one start

ZRAM Swap

Zram is more useful than disk swap. Install zram-init:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


apk add zram-init

# Adjust as needed. The default config creates swap and /tmp on zram.
#
# I usually set swap zram to roughly the same size as RAM and use lz4
# for speed. Because zram compresses memory, that does not mean it
# immediately consumes that much physical RAM.
#
# To set swap zram to the same size as RAM, remove the default
# `size0=512M` line and use:
# size0=`LC_ALL=C free -m | awk '/^Mem:/{print int($2)}'`
#
# To use lz4:
# algo0=lz4
#
# The /tmp zram can be left untouched. If you want a smaller /tmp, use
# something like:
# size1=`LC_ALL=C free -m | awk '/^Mem:/{print int($2/4)}'`
#
# Remove `blck1=1024` if your default config has it. The minimum block
# size is 4 KiB, so 1024 bytes is not valid.
#
# If /tmp is already mounted as tmpfs in /etc/fstab, remove that line
# when /tmp is provided by zram-init.
vi /etc/conf.d/zram-init

rc-update add zram-init boot
rc-service zram-init start

These sysctls make the kernel more willing to use compressed swap:

1
2
3
4
5
6
7
8


cat <<'EOF' > /etc/sysctl.d/01-zram.conf
vm.swappiness = 180
vm.watermark_boost_factor = 0
vm.watermark_scale_factor = 125
vm.page-cluster = 0
EOF

sysctl -p /etc/sysctl.d/01-zram.conf

Cloud-init

cloud-init is convenient if the image will be imported into a cloud provider, but it pulls in Python and is not small.

1
2
3


apk add cloud-init
# Read the installation notes displayed by apk. It will show which
# cloud-init services should be enabled for your target environment.

Read the package message after installation and enable the OpenRC services it lists. If you want automatic partition growth on first boot, also install:

1

apk add cloud-utils-growpart

For the smallest image, skip cloud-init and inject network configuration another way.

Expand Root After Restoring To A Larger Disk

If you restore the QCOW2 to a larger disk, grow the root partition and then grow Btrfs.

Run:

1

fdisk /dev/vda

Inside fdisk:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


p # print; write down the current start sector of vda2, usually 133120
d # delete partition
2 # select vda2

n # new partition
p # primary
2 # partition number 2
133120 # use the exact old start sector
<ENTER> # default end, use the rest of the disk

> If asked:
> Partition #2 contains a btrfs signature.
> Do you want to remove the signature? [Y]es/[N]o:

N # DO NOT remove the signature

p # verify vda2 starts at the same sector and uses the rest of the disk
w # write changes

Ask the kernel to reread the partition table:

1
2


apk add parted
partprobe /dev/vda || reboot

If you had to reboot, continue after the reboot:

1
2


btrfs filesystem resize max /
df -h /

I prefer this manual method over blindly using a grow tool because it actually expands and aligns, while some grow tools may not expand the partition to be 4K aligned.

BBR Congestion Control

For forwarding boxes, I usually enable BBR:

1
2
3
4
5
6


cat <<'EOF' > /etc/sysctl.d/02-bbr.conf
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
EOF

sysctl -p /etc/sysctl.d/02-bbr.conf

Verify:

1

sysctl net.ipv4.tcp_congestion_control

Podman

If I need containers on a tiny VM, I prefer Podman over Docker because the entire stack (no containerd, crun instrad of runc, lighter daemon).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


apk add podman

# Keep the overlay storage driver. It is usually more stable and has
# better performance for containers, even though the underlying
# filesystem is Btrfs.
#
# Limit container logs to 1M so they do not fill the disk.
sed -i 's/^#log_size_max =.*/log_size_max = 1048576/' /etc/containers/containers.conf

# Enable cgroups v2 for better compatibility with modern container runtimes.
rc-update add cgroups default
rc-service cgroups start

# Start containers that have a restart policy such as always or unless-stopped.
rc-update add podman default
rc-service podman start

If you want Docker CLI and Compose compatibility against the Podman socket:

1
2
3
4


apk add docker-cli docker-cli-compose

docker context create podman --docker "host=unix:///run/podman/podman.sock"
docker context use podman

I do not use podman-docker or podman-compose here; the Docker CLI is closer to what my existing Compose files expect.

UFW

If the provider does not give you a firewall, configure one before exposing the VM:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73


apk add ip6tables ufw
# Start ufw later, after the allow rules are in place.

# Default policies
ufw default deny incoming
ufw default allow outgoing

# Allow loopback
ufw allow in on lo

# -------------------------
# SSH
# -------------------------

# SSH port; change this if you use a non-standard SSH port.
ufw allow 22/tcp comment 'SSH'

# Optional: rate limit SSH brute force.
# ufw limit 22/tcp

# -------------------------
# Private / intranet ranges
# -------------------------

# RFC1918 IPv4
ufw allow from 10.0.0.0/8
ufw allow from 172.16.0.0/12
ufw allow from 192.168.0.0/16

# Optional CGNAT range
# ufw allow from 100.64.0.0/10

# Optional IPv6 ULA
ufw allow from fc00::/7

# Also allow forwarding to RFC1918 and ULA ranges if this server is a
# gateway or VPN server, or if you use Podman containers with published
# ports. Published container ports usually pass through DNAT and are
# handled by forward rules.
#
# Note that if you are using Podman containers with published ports,
# these route rules can allow access to all published ports even if the
# input rules only allow selected ports. If that is not what you want,
# only allow specific forwarded ports here.
# ufw route allow to 10.0.0.0/8
# ufw route allow to 172.16.0.0/12
# ufw route allow to 192.168.0.0/16
# ufw route allow to fc00::/7

# -------------------------
# Public services
# -------------------------

# Web
# ufw allow 80/tcp
# ufw allow 443/tcp
# ufw allow 443/udp

# Example app ports
# ufw allow 3000/tcp
# ufw allow 9090/tcp

# -------------------------
# Logging
# -------------------------

ufw logging low

# Enable firewall
ufw enable

rc-update add ufw default
rc-service ufw start

If this machine routes VPN, overlay network, or container traffic, add explicit ufw route allow ... rules for the private ranges you actually need.

sshguard

I avoid Fail2ban on this image because Python is too heavy for the target machine. sshguard is small and good enough.

1

apk add sshguard nftables

Do not blindly start the nftables service before checking its default ruleset from a console. On one of my Alpine images, doing that installed a default deny ruleset and locked out remote SSH. I let sshguard manage its own nftables sets instead.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


# IMPORTANT: Do not run `rc-update add nftables && rc-service nftables start`
# unless you have checked the ruleset from a console. We will let sshguard
# manage its nftables sets.

mkdir -p /var/lib/sshguard

cat <<'EOF' > /etc/sshguard.conf
# Full path to backend executable. Required; there is no default.
BACKEND='/usr/libexec/sshg-fw-nft-sets'

# Space-separated list of log files to monitor.
FILES='/var/log/messages'

# Block attackers when their cumulative attack score exceeds THRESHOLD.
# Most attacks have a score of 10.
THRESHOLD=20

# Block attackers for initially BLOCK_TIME seconds after exceeding THRESHOLD.
# Subsequent blocks increase by a factor of 1.5.
BLOCK_TIME=180

# Remember potential attackers for up to DETECTION_TIME seconds before
# resetting their score.
DETECTION_TIME=3600

# Permanently blacklist attackers when their cumulative score exceeds threshold.
BLACKLIST_FILE=100:/var/lib/sshguard/blacklist.db

# Size of IPv6 subnet to block. Defaults to a single address.
IPV6_SUBNET=48

# Size of IPv4 subnet to block. Defaults to a single address.
IPV4_SUBNET=24
EOF

rc-update add sshguard default
rc-service sshguard start

Common Tools

This is no longer minimal, but it makes Alpine feel closer to a general-purpose server if you find busybox too limited:

1
2


apk add bash coreutils findutils tar grep sed gawk diffutils procps util-linux shadow curl wget iproute2 bind-tools gcompat pciutils
apk add netcat-openbsd socat tcpdump iftop iptraf-ng ethtool traceroute zsh git htop tmux vim less jq iperf3 sysstat rsync file

And other tools

1

apk add bat dufs gdu lsd yazi

Final Notes

The most important part of this setup is the partitioning discipline. On a 1 GB disk, a lazy 512 MB /boot, a swap partition, or unbounded logs will consume the entire machine faster than the application does.

With a 64 MB /boot, compressed Btrfs root, no disk swap, and a few periodic cleanup jobs, Alpine stays usable even on tiny cloud disks. It is not luxurious, but it is enough for a forwarding node or a small single-purpose service.

Bundling macOS Dynamic Libraries

Mon, 08 Jun 2026 12:25:00 +0800

I recently tried to build QEMU on macOS and make the result portable enough that it could still run after uninstalling the Homebrew packages used during the build, or I can move the whole QEMU directory to another machine. I don’t want the Homebrew build because it installs too many dependencies on the system, and I want a more self-contained bundle, not a system-wide installation.

The short version: fully static linking on macOS is not really the right target. macOS binaries still dynamically link against Apple system libraries such as libSystem. But for tools like QEMU, what I really wanted was not a “pure static binary”; I wanted a self-contained directory like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


_qemu-11.0.1/
├── bin/
│ ├── qemu-img
│ ├── qemu-system-x86_64
│ └── ...
└── lib/
 ├── libglib-2.0.0.dylib
 ├── libgobject-2.0.0.dylib
 ├── libintl.8.dylib
 ├── libpcre2-8.0.dylib
 └── ...

The binaries in bin/ should load their non-system dependencies from ../lib (note that it’s relative, not absolute, so it will work even if the whole directory is moved), instead of from /opt/homebrew.

This post records the process, using QEMU as the example.

The original problem

I built QEMU roughly like this:

1
2
3
4
5
6


# You should refer to https://wiki.qemu.org/Hosts/Mac for the latest build instructions. This is just an example.
brew install libffi gettext glib pkg-config zstd # may not be the exact set of dependencies you need

./configure --prefix="$HOME/bin/_qemu-11.0.1"
make -j"$(sysctl -n hw.ncpu)"
make install

The build worked. But after uninstalling some Homebrew dependencies, running qemu-img failed:

1
2
3
4


dyld[82783]: Library not loaded: /opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib
 Referenced from: /Users/charlie/bin/_qemu-11.0.1/bin/qemu-img
 Reason: tried: '/opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib' (no such file)
zsh: abort qemu-img

This means the installed QEMU binary still contains an absolute dependency path:

1

/opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib

We can confirm that with:

1

otool -L "$PREFIX/bin/qemu-img"

Example output:

1
2
3
4
5


qemu-img:
 /opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib (compatibility version 8801.0.0, current version 8801.0.0)
 /opt/homebrew/opt/zstd/lib/libzstd.1.dylib (compatibility version 1.0.0, current version 1.5.7)
 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0)
 ...

The /usr/lib and /System/Library entries are fine. They are macOS system libraries. The /opt/homebrew/... entries are the ones that make the binary depend on my local Homebrew installation.

Why not just build QEMU fully static?

On Linux, the natural thought is “just build a static binary”.

On macOS, that is not usually how things work. macOS does not support the same style of fully static userland binary that Linux users often expect. You can try to statically link some third-party libraries, but the final program will still dynamically link to Apple system libraries.

So the practical goal is:

Bundle third-party .dylib dependencies next to the program, and rewrite Mach-O load commands so the binary loads those local copies.

This is similar in spirit to what many .app bundles do, but here I wanted a plain CLI directory layout.

The tools involved

macOS Mach-O binaries can be inspected and patched with these tools:

1
2
3
4


otool -L <file>
install_name_tool -change <old> <new> <file>
install_name_tool -id <new-id> <dylib>
install_name_tool -add_rpath <rpath> <file>

otool -L shows dynamic library load commands.

For example:

1

otool -L "$PREFIX/bin/qemu-img"

might show:

1
2
3
4
5


qemu-img:
 /opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib (compatibility version 8801.0.0, current version 8801.0.0)
 /opt/homebrew/opt/zstd/lib/libzstd.1.dylib (compatibility version 1.0.0, current version 1.5.7)
 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0)
 ...

Those can be rewritten with:

1
2
3
4


install_name_tool -change \
 /opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib \
 '@executable_path/../lib/libglib-2.0.0.dylib' \
 "$PREFIX/bin/qemu-img"

@executable_path means “the directory containing the executable being launched”.

So it will try to find libglib-2.0.0.dylib in ../lib/ relative to the executable, no longer looking in /opt/homebrew.

First attempt: patch direct dependencies only

My first attempt was simple:

1
2
3
4
5


otool -L "$PREFIX"/bin/* \
 | grep /opt/homebrew \
 | sort \
 | uniq \
 | awk '{print $1}'

This found direct Homebrew dependencies used by QEMU binaries, such as:

1
2
3
4
5


/opt/homebrew/opt/glib/lib/libgio-2.0.0.dylib
/opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib
/opt/homebrew/opt/glib/lib/libgmodule-2.0.0.dylib
/opt/homebrew/opt/glib/lib/libgobject-2.0.0.dylib
/opt/homebrew/opt/zstd/lib/libzstd.1.dylib

Then I copied those libraries into $PREFIX/lib and patched the binaries.

This fixed some errors, but not all of them.

Running qemu-img then failed with a different error:

1
2


dyld[97126]: Library not loaded: /opt/homebrew/opt/gettext/lib/libintl.8.dylib
 Referenced from: /Users/charlie/bin/_qemu-11.0.1/lib/libglib-2.0.0.dylib

So I realized that it is not enough to patch only the binaries. The copied .dylib files have their own dependencies too.

In this case:

1
2
3


qemu-img
 -> libglib-2.0.0.dylib
 -> libintl.8.dylib

libintl.8.dylib was not directly referenced by qemu-img. It was a transitive dependency of libglib.

The correct model

The dependency graph looks like this:

1
2
3
4
5
6
7


bin/qemu-img
 -> /opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib
 -> /opt/homebrew/opt/zstd/lib/libzstd.1.dylib

lib/libglib-2.0.0.dylib
 -> /opt/homebrew/opt/gettext/lib/libintl.8.dylib
 -> /opt/homebrew/opt/pcre2/lib/libpcre2-8.0.dylib

So the bundling process must be recursive:

Scan Mach-O files in bin/.
Find its /opt/homebrew dependencies.
Copy those .dylib files into lib/.
Patch the binaries to refer to the bundled copies (../lib).
Scan the copied .dylib files.
Copy and patch their dependencies too.
Repeat until no /opt/homebrew dependencies remain.

Which paths should be used?

For executables in bin/, I use:

1

@executable_path/../lib/libfoo.dylib

For dylibs inside lib/, I use:

1

@loader_path/libfoo.dylib

@executable_path is relative to the main executable being launched.
@loader_path is relative to the Mach-O file that is doing the loading. For dependencies between dylibs, this is what we want:
1 2

lib/libglib-2.0.0.dylib -> @loader_path/libintl.8.dylib

Since both files are in the same lib/ directory, this resolves correctly.

The final bundling script

I ended up writing a small Python script called macho-bundle-deps. You can find it here: https://github.com/charlie0129/dotfiles/blob/f15791054e517f6b5afc892312f1db73f331d475/bin/darwin/macho-bundle-deps. This is a permanent link to a specific commit, so you may want to check the latest version in the repository.

Usage:

1
2
3
4
5
6


PREFIX="$HOME/bin/_qemu-11.0.1"

macho-bundle-deps \
 --lib-dir "$PREFIX/lib" \
 --prefix /opt/homebrew \
 "$PREFIX"/bin

Or, from inside the QEMU prefix:

1
2
3


cd "$HOME/bin/_qemu-11.0.1"

macho-bundle-deps --lib-dir lib bin

The script does the following:

finds Mach-O files from the input paths
scans their dependencies with otool -L
copies matching dependencies into --lib-dir
patches executable dependencies to @executable_path/../lib/...
patches bundled dylib dependencies to @loader_path/...
patches copied dylib IDs with install_name_tool -id
recursively processes newly copied dylibs
reports an error if matching absolute dependency paths remain

Verifying the result

After running the bundler, I verify that no Homebrew paths remain:

1

otool -L "$PREFIX"/bin/* "$PREFIX"/lib/*.dylib | grep /opt/homebrew

Now I can uninstall the dependencies from Homebrew, and the bundled QEMU still works:

1
2


brew uninstall libffi gettext glib
qemu-img --help

Notes and limitations

This is not the same as producing a fully static binary.

The result still depends on macOS system libraries, which is normal:

1
2


/usr/lib/libSystem.B.dylib
/System/Library/Frameworks/...

It also does not magically make a binary portable across all macOS versions or CPU architectures. A binary built on Apple Silicon is still an arm64 Mach-O binary unless built otherwise.

This approach is mainly useful for making a local CLI tool directory self-contained with respect to Homebrew dependencies.

Why copying symlinks should be avoided

Homebrew often has dylib symlinks like:

1

libintl.dylib -> libintl.8.dylib

When bundling, prefer copying the symlink target, not the symlink itself.

In shell, that means:

1

cp -L source.dylib "$PREFIX/lib/"

In Python, shutil.copy2(..., follow_symlinks=True) does the same thing.

This avoids ending up with a bundled symlink pointing to a non-existent file.

Final layout

After bundling, the QEMU directory looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


_qemu-11.0.1/
├── bin/
│ ├── qemu-img
│ ├── qemu-io
│ ├── qemu-nbd
│ ├── qemu-system-aarch64
│ ├── qemu-system-x86_64
│ └── ...
└── lib/
 ├── libffi.8.dylib
 ├── libgio-2.0.0.dylib
 ├── libglib-2.0.0.dylib
 ├── libgmodule-2.0.0.dylib
 ├── libgobject-2.0.0.dylib
 ├── libintl.8.dylib
 ├── libpcre2-8.0.dylib
 └── libzstd.1.dylib

Now qemu-img no longer cares whether Homebrew’s glib, gettext, pcre2, or zstd packages are installed.

Conclusion

The main lesson is that macOS dependency bundling is graph traversal, not a one-shot patch.

Patching only the top-level binaries is not enough. You also need to patch the dylibs that you copied, and then patch the dylibs that those dylibs depend on.

The final strategy is:

1

scan -> copy -> patch -> scan copied dylibs -> repeat

For executables:

1
2


/opt/homebrew/.../libfoo.dylib
 -> @executable_path/../lib/libfoo.dylib

For bundled dylibs:

1
2


/opt/homebrew/.../libbar.dylib
 -> @loader_path/libbar.dylib

And for the dylib’s own install name:

1
2


/opt/homebrew/.../libfoo.dylib
 -> @loader_path/libfoo.dylib

(Chinese Only) 华为 AirEngine 5773-21 踩坑记录

Sat, 13 Dec 2025 16:00:00 +0800

背景

最近老家重建，家里需要重新布置无线网络。考虑到自建别墅面积比较大，单纯使用单个无线路由器覆盖不够，因此多个 AP 肯定是需要的。家里在装修的时候就每个房间吊顶中有留超六类线缆，这种情况对 AP 来说是非常友好的，因此决定使用企业级无线接入点（AP）来组建无线网络。由于华为系列的 AP 在国内使用量较大，闲鱼上非常好捡公司下架下来的二手设备，因此考虑华为系列的 AP。这些企业级 AP 并不需要担心二手问题，他们的质量是消费级产品无法比拟的，稳定性也是非常强。

目前是 2025 年 12 月，虽然在国内 Wi-Fi 7 （802.11be）的 6 GHz 频段还没批准无法使用（损失不少性能），但是上一下 Wi-Fi 7 追一下新也没什么问题。华为 Wi-Fi 7 AP 是 AirEngine xx7x 产品，我闲鱼花了 500 多，激情下单了一台 AirEngine 5773-21 （有一个 2.5 Gbe PoE 口）。先买一台测测，为之后全屋部署积累经验，如果好用，再给每个房间买一台（后续更新）。

回来一查，这设备保修到 2028 年 3 月，现在才 2025 年 12 月，还有快 3 年的保修期，很新啊。

我目前的打算是不购买单独的 AC 控制器（AC 用于统一管理其他 AP ），因为我没有几十上百个 AP 需要管理。对于只需管理十几台 AP 来说，AC 控制器的成本和复杂度都不划算。因此打算使用华为 AP 自带的 Leader AP 模式组网（按道理华为的 AirEngine 系列大部分都是支持的，但是对于我买的 AirEngine 5773-21 来说，这里是个大坑，后面会讲）。这种模式下，Leader AP 可以担任 AC 的角色，统一管理其他 FIT AP ，每个 Leader AP 具体能管理几台 FIT AP 可以上华为 Info-Finder 上查询（基本上都大于 16 台）。

（组网方案后续更新，包括多 AP 的组网方案， VLAN 配置 IOT 设备隔离等）

首次启动

插上网线（注意，如果使用 PoE 的话需要满足 802.3at ，否则可能降速，或者使用 DC 供电）

注意，开机过程中状态灯常亮，等到状态灯快速闪烁说明开机完成等待配置。

现在华为的 AP 默认会用 DHCP 获取 IP 地址，因此只需根据 AP 的 MAC 地址（在 AP 背后会写）在 DHCP 服务器上找到对应的 IP 地址，就可以知道 AP 目前的 IP 地址了

然后浏览器打开该 IP 即可登录 Web 管理界面（并不）。

踩坑

然后浏览器打开该 IP 即可登录 Web 管理界面（吗？？？？），我发现并不能。

按道理华为 Wi-Fi 6 系列（比如 AirEngine 5761 ）都是可以直接登录 Web 管理页面的，Wi-Fi 7 更新不应该不行，而且官方 AirEngine 5773-21 的彩页中也说可以用做 FAT AP / Leader AP ，怎么回事？

我后面用 SSH 登录设备（后面会说怎么登录），没有说当前是 FIT 还是 FAT 模式（老的 AP 会在 SSH 登录的时候就提示）。而且这个命令行看的我一脸懵，老的 Wi-Fi 6 AP 都是直接 system-view 就能配置，然后命令也是传统的那一套。这新的 AP 一上来就是个 MDCLI> 提示符（一看就是新搞的东西，里面的命令也完全变了，我根本没时间去学习这些新命令。（后面才知道，必须升级系统后，用 switch cli 从 MC-CLI 切换到传统 CLI 模式，才能用老的命令行方式配置，但是建议还是用新的 MD-CLI ，因为传统 CLI 里面基本没啥功能了）

看了一圈文档，说可以输入 edit-config 修改配置，然后发现提示没有权限。WTF？我管理员还能没权限？后面才知道，原来没有权限其实代表当前 AP 在 FIT 模式下（这报错给用户带来多少困扰），必须切换到 FAT 模式才有权限修改配置。那么问题来了，我都没权限做任何配置，我怎么切换到 FAT 模式（况且我也没找到切换 FAT 模式的命令）？

后面折腾一圈才发现（各种根据文档中的蛛丝马迹去猜），是我当前固件版本太老了（V600R023C10），华为并没有在早期固件中实现 FAT AP / Leader AP 功能（设备都发布了，功能还没做好是吧），必须升级到 V600R024C10 及以上版本才有该功能，写文章时最新版本是 V600R025C00 。OK，问题找到，接下来就是升级固件了。

但是华为是出了名的不给资料，比如我想下载新版本固件，就算我注册了账号、也绑定了 AP 的序列号、也给了华为公司名称 + 设备序列号也审核通过了 AirEngine 5773-21 的资料和软件下载权限，但是你还是下不到 AirEngine 5773-21 的固件，华为官网上根本没有提供任何一个版本固件的下载链接（截至 2025 年 12 月），你只能下到补丁（这还是从其他 AP 找的补丁，刚好能用到 AirEngine 5773-21 上）。

最后我只能在闲鱼花钱上找有权限的人代下的固件，版本号 V600R024C10 ，也就是第一版支持 FAT AP / Leader AP 功能的固件。本来我想下载当前最新的 V600R025C00 ，但是我没找到人能下载这个版本，遂放弃， V600R024C10 也不是不能用。

首次设备设置

可以按住设备旁的 Default 按钮数十秒来重置设备。观察指示灯，变成常亮表示成功。重置成功后会重启，开机过程中状态灯常亮，等到状态灯快速闪烁说明开机完成等待配置。

你需要 SSH 上去 ssh admin@<AP_IP_ADDRESS> ，默认密码是 admin@huawei.com 。首次登录会提示你修改密码，按照提示修改后会登出，下次使用新密码再登录即可。

查看当前版本

1
2
3


[admin@HUAWEI]
MDCLI> display system/system-info/software-name
"AirEngineX773_V600R023C10SPC200.cc" # 当前版本

我目前的版本是 V600R023C10SPC200 ，也就是 V600R023C10 （后面的 SPC200 是补丁），需要升级到 V600R024C10 及以上版本才有 FAT AP / Leader AP 功能。

固件升级

首先需要找台机器开启 FTP 服务端，设置好用户名密码，打开读写权限（macOS App Store 中有个 QuickFTP 还是很好用的），到时候 AP 需要通过 FTP 上传和下载固件。

然后你需要有一个 V600R024C10 及以上版本的固件文件，放到 FTP 的根目录下面。

需要使用的固件名称通常长这样，两者都行：

AirEngineX773_V600R024C10.cc （不带补丁的基础固件）
AirEngineX773_V600R024C10SPC100.cc （带补丁的完整固件）

其中：

AirEngineX773：设备型号，比如 AirEngine 5773
V600R024C10：版本号，R 或者 C 之后的数字越大，版本越新。
SPC100：补丁号，可选。注意，不要下成 SPH 的热补丁了，补丁通常长这样 AirEngineX773_V600R024C10SPH150.pat（有 SPH 字样，且结尾是 .pat ），需要完整的固件（.cc 结尾）。

备份当前固件

SSH 登录设备操作

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


[admin@HUAWEI]
MDCLI> ftpc-transfer-file

[(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> command-type put

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> local-file-name AirEngineX773_V600R023C10SPC200.cc # 文件名与当前 AP 版本一致，查询：display system/system-info/software-name

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> remote-file-name AirEngineX773_V600R023C10SPC200.cc # 与上面保持一致即可

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> server-ipv4-address 192.168.213.54 # FTP 服务器 IP 地址

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> server-port 2121 # FTP 服务器端口

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> user-name xxxx # FTP 用户名

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> password # FTP 密码
Enter password:
Confirm password:

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> emit # 提交上传任务
{
 "huawei-ftpc:transfer-id": 3 # 可以用这个 ID 查看任务状态
}

[admin@HUAWEI]
MDCLI> display ftpc/transfer-tasks # 检查是否备份成功
{
 "transfer-task": [
 {
 "transfer-id": 3,
 "command-type": "put",
 "server-address": "192.168.213.54",
 "server-port": 2121,
 "local-file-name": "AirEngineX773_V600R023C10SPC200.cc",
 "remote-file-name": "AirEngineX773_V600R023C10SPC200.cc",
 "status": "succeeded", # 表示成功
 "percentage": 100
 }
 ]
}

你应该可以在 FTP 服务器上看到备份下来的固件文件了。

下载新固件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49


[admin@HUAWEI]
MDCLI> download-upgrade-package

[(x)admin@HUAWEI]/download-upgrade-package
MDCLI> base-software-directory AirEngineX773_V600R024C10SPC100.cc # 新固件文件名，注意要和 FTP 服务器上一致

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> server-ip 192.168.213.54 # FTP 服务器 IP 地址

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> server-port 2121 # FTP 服务器端口

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> transfer-protocol ftp # 传输协议

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> user-name xxxx # FTP 用户名

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> password # FTP 密码
Enter password:
Confirm password:

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> emit # 提交下载任务

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> display software/download-result/ # 查看下载状态
{
 "file-name": "AirEngineX773_V600R024C10SPC100.cc",
 "status": "succeeded", # 表示下载成功
 "percentage": 100
}

[admin@HUAWEI]
MDCLI> display file-operation/ # 查看文件列表，确认新固件在设备上
{
 "dir": [
...
 {
 "file-name": "AirEngineX773_V600R024C10SPC100.cc", # 新固件文件
 "dir-name": "backup:/",
 "attribute": "-rw-",
 "modify-time": "2024-11-12T12:05:04Z",
 "size": 52394852
 }
...
 ]
}

设置新固件为启动固件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


[admin@HUAWEI]
MDCLI> startup-by-mode name AirEngineX773_V600R024C10SPC100.cc # 设置启动固件，注意文件名要和上面一致

[admin@HUAWEI]
MDCLI> display cfg/startup-infos
{
 "startup-info": [
 {
 "position": "0",
 "configed-system-software": "AirEngineX773_V600R023C10SPC200.cc",
 "current-system-software": "AirEngineX773_V600R023C10SPC200.cc",
 "next-system-software": "AirEngineX773_V600R024C10SPC100.cc", # 确认新固件已设置为下次启动固件
 "current-cfg-file": "",
 "next-cfg-file": "",
 "current-patch-file": "NULL",
 "next-patch-file": "NULL"
 }
 ]
}

注意，要是你之前安装了 patch ，可能会导致新系统安装不上，例如

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


[admin@HUAWEI]
MDCLI> display cfg/startup-infos
{
 "startup-info": [
 {
 "position": "0",
 "configed-system-software": "AirEngineX773_V600R023C10SPC200.cc", # 是的，华为拼错单词了
 "current-system-software": "AirEngineX773_V600R023C10SPC200.cc",
 "next-system-software": "AirEngineX773_V600R023C10SPC200.cc", # 发现还是老的版本
 "current-cfg-file": "",
 "next-cfg-file": "",
 "current-patch-file": "AirEngineX773_V600R023SPH151.pat", # 已安装 patch
 "next-patch-file": "AirEngineX773_V600R023SPH151.pat"
 }
 ]
}

这时候你需要把 patch 给清掉：

1
2
3


[admin@HUAWEI]
MDCLI> delete-patch delete-type all
......

然后再设置启动固件即可。

重启设备

1
2
3
4


[admin@HUAWEI]
MDCLI> reboot
Warning: This operation will reboot the device.
Are you sure you want to continue? [Y(yes)/N(no)]:y

等设备起来，浏览器访问设备 IP 地址，应该就可以看到 Web 管理界面了。

选择右上角 FIT 按钮，改成 FAT 模式即可使用 Leader AP 功能，重启后即可变成 Leader AP ，可以管理其他 FIT AP 了。

安装补丁（Patch）

补丁是在基础版本上修复 bug 或者增加小功能的。补丁一般靠自己注册完设备就能在官网下到，下载补丁的时候，注意产品型号不需要选择 AirEngine 5773-21 ，因为太新了，华为官网上甚至没有这个型号。直接选择全部，只要版本对的上即可，比如我当前基础版本是 V600R024C10 ，那么补丁就需要选择 V600R024C10SPH181 这种补丁，表示在 V600R024C10 基础版本上打的热补丁 SPH181 ，数字越大表示补丁越新。补丁文件名通常长这样 AirEngineX773_V600R024C10SPH181.pat ，注意是 .pat 结尾。

下载后，放到 FTP 服务器根目录下，然后 SSH 登录设备，执行以下命令安装补丁：

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101


[admin@HUAWEI]
MDCLI> ftpc-transfer-file

[(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> command-type get

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> local-file-name AirEngineX773_V600R024C10SPH181.pat # 补丁文件名，注意和 FTP 服务器上一致

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> remote-file-name AirEngineX773_V600R024C10SPH181.pat # 与上面保持一致即可

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> server-ipv4-address 192.168.213.54 # FTP 服务器 IP 地址

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> server-port 2121 # FTP 服务器端口

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> user-name xxx # FTP 用户名

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> password # FTP 密码
Enter password:
Confirm password:

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> emit # 提交下载任务
{
 "huawei-ftpc:transfer-id": 1
}

[admin@HUAWEI]
MDCLI> display ftpc/transfer-tasks/ # 查看任务状态
{
 "transfer-task": [
 {
 "transfer-id": 1,
 "command-type": "get",
 "server-address": "192.168.213.54",
 "server-port": 2121,
 "local-file-name": "AirEngineX773_V600R024C10SPH181.pat",
 "remote-file-name": "AirEngineX773_V600R024C10SPH181.pat",
 "status": "succeeded", # 表示成功
 "percentage": 100
 }
 ]
}

[admin@HUAWEI]
MDCLI> load-patch name AirEngineX773_V600R024C10SPH181.pat load-type run # 安装补丁，注意文件名与需要安装的补丁一致

[admin@HUAWEI]
MDCLI> display patch/operation-schedules # 查看安装进度
{
 "operation-schedule": [
 {
 "phase": "load-patch",
 "status": "successful",
 "schedule": 100 # 等待进度 100%
 },
 {
 "phase": "delete-patch",
 "status": "not-started",
 "schedule": 0
 },
 {
 "phase": "startup-next-patch",
 "status": "not-started",
 "schedule": 0
 },
 {
 "phase": "reset-startup-patch",
 "status": "not-started",
 "schedule": 0
 }
 ]
}

[admin@HUAWEI]
MDCLI> display patch/patch-infos
{
 "patch-info": [
 {
 "name": "AirEngineX773_V600R024C10SPH181.pat", # 验证补丁安装成功
 "version": "V600R024C10SPH181",
 "state": "running",
 "runtime": "2025-12-13T21:39:48+08:00",
 "path": "/",
 "operations": {
 "operation": [
 {
 "position": "0",
 "position-type": "MPU",
 "upgrade-mode": "reset-board" # 某些补丁需要重启设备才能生效
 }
 ]
 }
 }
 ]
}

你也可以在 Web 管理界面上查看补丁是否安装成功。

Bonus: 启用 IPv6

我发现 AP 下面的设备获取不到 IPv6 地址（我内网中 IPv6 配置都是正确的），基本上肯定是在 AP 导致的 IPv6 地址无法下发。

后面了解华为 AP 默认不开 IPv6 报文转发。原因是：华为认为 IPv4 是主流，在 IPv4 网络中，如果存在较多 IPv6 协议报文，会影响无线网络性能，也会损耗设备的 CPU 处理能力。因此在纯 IPv4 网络中，可以通过不处理 IPv6 无线报文来提高 IPv4 网络性能。

不过，我就是要用 IPv6 ，因此要去把 WLAN 处理 IPv6 报文的功能打开（ Web UI 没有，要命令行开），SSH 登录设备，执行以下命令：

官方文档里面其实有这个，但是我找半天没找到，在：“参考-MD-CLI配置参考-WLAN配置-WLAN用户管理” 里面。不是，你把 IPv6 相关配置放在“用户管理”里面？这跟用户管理有啥关系？

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


[admin@HUAWEI]
MDCLI> edit-config

[(gl)admin@HUAWEI]
MDCLI> wlan-sta-access

[(gl)admin@HUAWEI]/wlan-sta-access
MDCLI> sta-ipv6-switch true

[*(gl)admin@HUAWEI]/wlan-sta-access
MDCLI> commit

之后你的设备应该就能正确收发 IPv6 报文了。

结语

这告诉我们， AirEngine 5773-21 这种 Wi-Fi 7 新设备，还是太新了，所有的坑都得自己踩。与 Wi-Fi 6 的设备比如 AirEngine 5761 相比网上大把教程还是差太远了。不过也算是积累了一些经验，后续再买多台 AP 的时候就不会再踩这些坑了。

这次单台 AP 就放北京用了，年底回老家再配置多 AP 组网方案。

Set Up ZeroTier Moon on Non-Standard Ports

Thu, 27 Nov 2025 10:20:00 +0800

To install ZeroTier, one would typically allow port 9993/UDP on their firewall. However, in certain scenarios, you may need to run a ZeroTier moon on non-standard ports due to network restrictions (e.g. behind NAT) or conflicts with other services. This guide will walk you through the steps to set up a ZeroTier moon on non-standard ports.

Setup Relays (Moon Nodes) on Non-Standard Ports

Install ZeroTier like usual:

1
2
3


curl -s https://install.zerotier.com | sudo bash
# and join a network (if your moon also acts as a client)
zerotier-cli join <network_id>

Setup moon:

1

zerotier-idtool initmoon /var/lib/zerotier-one/identity.public >>/var/lib/zerotier-one/moon.json

Here is the important part: edit the moon.json file to specify the desired non-standard ports. Open the file with your preferred text editor:

1

vim /var/lib/zerotier-one/moon.json

Modify the stableEndpoints section to include your public IP address along with the desired non-standard port. If you are behind a NAT, use your router’s public IP address and forward the same port from your server to your router. The format should be IP_ADDRESS/PORT.

For example, if you want to use port 14999, change the line to:

1

"stableEndpoints": ["xxx.xxx.xxx.xxx/14999"]

Generate the moon configuration (this is what clients will use to connect to your moon):

1

zerotier-idtool genmoon /var/lib/zerotier-one/moon.json

You should have one file that looks like *.moon in your current dir. Move the generated moon file to the ZeroTier directory:

1
2


mkdir -p /var/lib/zerotier-one/moons.d/
mv *.moon /var/lib/zerotier-one/moons.d/

If your moon also acts as a client, change the client configuration to use the non-standard port. Edit /var/lib/zerotier-one/local.conf:

1
2
3
4
5


{
 "settings": {
 "primaryPort": 14999
 }
}

Restart the ZeroTier service to apply the changes:

1

systemctl restart zerotier-one

Setup Clients (Leaf Nodes)

On the client side, you have to change the default port as well. Yes, the client’s default port (9993) must match the moon’s port. Otherwise, they won’t be able to communicate in my tests.

After you installed ZeroTier and joined the network, edit /var/lib/zerotier-one/local.conf:

1
2
3
4
5


{
 "settings": {
 "primaryPort": 14999
 }
}

Copy /var/lib/zerotier-one/moons.d/*.moon from the moon server to the client machine’s /var/lib/zerotier-one/moons.d/ directory, so the client knows about the moon:

1

scp "user@moon-server:/var/lib/zerotier-one/moons.d/*.moon" /var/lib/zerotier-one/moons.d/

Restart the ZeroTier service on the client:

1

systemctl restart zerotier-one

Verification

You can verify that the moon is functioning correctly by checking zerotier-cli command on the client to see if it can connect to the moon.

1

zerotier-cli peers

You should see an entry for your moon with the correct non-standard port, similar to the example below (xxx.xxx.xxx.xxx/14999):

1
2
3
4
5
6
7
8
9


zerotier-cli peers
200 peers
<ztaddr> <ver> <role> <lat> <link> <lastTX> <lastRX> <path>
35c192ce9b 1.15.3 LEAF 287 DIRECT 11575 11575 2001:19f0:6001:2c59:beef:3d:6767:df71/21006
3cdfac522d 1.16.0 MOON 66 DIRECT 3660 3660 xxx.xxx.xxx.xxx/14999
778cde7190 - PLANET 287 DIRECT 44090 43802 2605:9880:400:c3:254:f2bc:a1f7:19/9993
cafe04eba9 - PLANET 287 DIRECT 44090 43802 84.17.53.155/9993
cafe80ed74 - PLANET 261 DIRECT 269315 43837 2a02:6ea0:c87f::1/9993
cafefd6717 - PLANET 246 DIRECT 299345 43845 2a02:6ea0:d368::9993/9993

If your <lat> is -1, it means the client cannot reach the moon. Double-check your configuration and ensure that the specified ports are open and correctly forwarded if behind a NAT.

Proxmox VE Shared GPU Installation Guide

Sat, 16 Aug 2025 20:49:00 +0800

Introduction

Most schools and universities in China have a limited number of GPU servers (often only one or two, or even none). This makes the management of GPU development machines quite different than corporation which have virtually unlimited GPU servers.

Let’s consider a normal lab environment with 12 students, each needing a GPU server for their projects, and we have one GPU server with 8 GPUs available.

If we give each student a dedicated GPU server, we would need 12 servers, this is impractical and costly.
If we let all the students share a single GPU server (all students have access to all resources), it would be very difficult to manage, as each student would need to install their own software and dependencies, leading to conflicts. Soon the GPU server will become a hot mess. Making each student use a different user account would not solve the problem, as they would still share the same OS and software environment, and will potentially break each other’s environment.
If we make use of IOMMU and PCIe passthrough, we can assign a virtual machine with one dedicated passthrough’d GPU to each student. This approach has almost no interference between students, as each student has their own OS and software environment, even with it’s own GPU. However, this leads to a lot of wasted resources, because most of the time the GPU is idle, and each student will only have access to one GPU at a time, even if all of the GPUs are idle.
If we use GPU virtualization, we can assign a virtual machine with a shared GPU to each student. This way, each student can use the GPU resources as needed, and the GPU can be shared among multiple students. This approach is more efficient and cost-effective, as it allows for better resource utilization. However, it requires a GPU that supports virtualization, such as NVIDIA’s vGPU or AMD’s MxGPU. Most consumer GPUs do not support virtualization, so this approach is not feasible for most schools and universities.

This is where containerization comes in. By using containerization, we can create a shared GPU development server that runs a container for each student. Each container can have its own software environment, and all (or some of) of the GPUs can be shared among multiple containers. This approach is more efficient and cost-effective, as it allows for better resource utilization, and does not require a GPU that supports virtualization. However, it should be noted that this approach is not as isolated as virtual machines, as all containers share the same kernel and GPU resources. Therefore, it is important to ensure that the containers are properly configured, and that the students are aware of the limitations and potential issues that may arise from sharing the same GPU resources (for example, if one student runs a GPU-intensive task, it may affect the performance of other students’ containers).

We will be using the CT (LXC Containers) in Proxmox VE to achieve this. Why not use Docker? Because Docker is meant for running applications, not for running full Linux distros. Although you can use something like sysbox to run a full Linux distro in Docker, i would still prefer to use LXC Containers, as they are built for this purpose and are already built into Proxmox VE, making it easier to manage and deploy.

Proxmox VE Installation

Download the latest Proxmox VE ISO from the official website. At the time of writing, the latest version is Proxmox VE 9.0. Use whatever method you prefer to install Proxmox VE, such as using a USB drive.

I will use IPMI of the server to install Proxmox VE, as it is the most convenient method for me. You can also use a monitor and keyboard to install Proxmox VE if you prefer.

Mount the Proxmox VE installation ISO as virtual media so we can boot into.

Now reboot the server into the virtual media. Supermicro motherboards lets you invode the boot menu using F11. Your motherboard may have a different shortcut.

Choose our CDROM virtual media.

Boot into the Proxmox install menu (I prefer terminal UI over GUI ones).

After you accepted the license, you will need to choose the target installtion disk. You should choose the boot SSDs on your server, not data drives. The drive in picture is 3*1.92TB SSDs in RAID 5.

Toggle Advanced options. Keep ext4, we will not be using ZFS. I know ZFS has CoW, snapshotting, compressing, checksums, and a ton of other features. But we will not use ZFS, because it will potentially cause some problems, e.g., high IO load when using RAID-Z zvols, extremely slow container start times when using ZFS storage driver in Docker, SSD write amplifications, low SSD random read/write performance, ZFS ARC not being given back to the OS as fast as needed on high memory pressure systems leading to OOM, and a ton of other issues that I previously encountered. To save me some trouble, I will use the battle-tested ext4.

Total size: keep as-is
Swap size: 0, we don’t need it, our memory is large enought (1TiB) and we will later use ZRAM as swap
Maximum root volume size: 100 (GiB), make it slightly larger so we can install things into the root volume, but not too large to occupy the data volume space.

Other options can be left empty.

Other install steps (Keyboard setup, root password) can proceed as you normally would do. Remember to set a complex root password.

Regarding to IP addresses:

If you are going to use a static IP address like I do, you can just set it here and forget about it.

If you are going to use a dynamic (DHCP) addresses, you should keep what’s already in here (static IP). You don’t need to change anything. The valid IP address should already be automatically discovered from DHCP servers and filled in. After the installation, you can change the static IP mode to DHCP mode by running vi /etc/network/interfaces and make the following edits:

1
2
3
4
5
6
7
8


 auto vmbr0
- iface vmbr0 inet static
+ iface vmbr0 inet dhcp # Enable DHCP and remove static IP.
- address 10.112.154.220/16
- gateway 10.112.0.1
 bridge-ports eno1
 bridge-stp off
 bridge-fd 0

After you enabled DHCP in the config, apply it by ifreload -a. Check if it takes effect by ip a show vmbr0, you should see something like valid_lft 7160sec preferred_lft 7160sec. If there is no xx sec inside it, your DHCP is malfunctioning.

1
2
3
4
5
6
7


root@z8:/etc/network# ip a show vmbr0
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
 link/ether 30:13:8b:6d:3a:a6 brd ff:ff:ff:ff:ff:ff
 inet 10.112.154.220/16 brd 10.112.255.255 scope global dynamic vmbr0
 valid_lft 7160sec preferred_lft 7160sec
 inet6 fe80::3213:8bff:fe6d:3aa6/64 scope link
 valid_lft forever preferred_lft forever

Proxmox VE Post-Installtion

Change APT Sources

In China, we don’t really have a good global Internet connection, so we will change APT source to mirrors in China.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# Change APT source mirrors
sed -i 's/deb.debian.org/mirrors.ustc.edu.cn/g' /etc/apt/sources.list.d/debian.sources
sed -i 's/security.debian.org/mirrors.ustc.edu.cn/g' /etc/apt/sources.list.d/debian.sources

# Change Ceph source mirrors
if [ -f /etc/apt/sources.list.d/ceph.sources ]; then
 CEPH_CODENAME=`ceph -v | grep ceph | awk '{print $(NF-1)}'`
 source /etc/os-release
 cat > /etc/apt/sources.list.d/ceph.sources <<EOF
Types: deb
URIs: https://mirrors.ustc.edu.cn/proxmox/debian/ceph-$CEPH_CODENAME
Suites: $VERSION_CODENAME
Components: no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF
fi

Remove enterprise repo and add no-subscription repo.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Remove enterprise sources
rm /etc/apt/sources.list.d/pve-enterprise.sources

# Add no-subscription sources
cat > /etc/apt/sources.list.d/pve-no-subscription.sources <<EOF
Types: deb
URIs: https://mirrors.ustc.edu.cn/proxmox/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF

Change CT Template Sources

Use mirrors in China.

1
2


sed -i.bak 's|http://download.proxmox.com|https://mirrors.ustc.edu.cn/proxmox|g' /usr/share/perl5/PVE/APLInfo.pm
systemctl restart pvedaemon

Stop Cluster Services

We are not using PVE clusters. Disable them. If you are using PVE clusters, you probably will not be reading this guide :P

1
2
3


systemctl disable --now pve-ha-crm.service
systemctl disable --now pve-ha-lrm.service
systemctl disable --now corosync.service

Install Common Tools

Useful tools that will be used regularly.

1

apt install htop sysstat vim sudo

Configure Shell

I like ZSH and my dotfiles, so I will use them.

1

apt install git zsh

1
2
3
4
5
6
7
8


cd
git clone --depth=1 https://github.com/charlie0129/dotfiles.git
cd dotfiles
./bootstrap.sh -f
chsh -s /usr/bin/zsh

# Run zsh and follow the instructions
zsh

Configure ZRAM Swap

1
2
3
4
5


git clone --depth=1 https://github.com/foundObjects/zram-swap.git
cd zram-swap
./install.sh
cd ..
rm -rf zram-swap

The default lz4 is a good balance between speed and compression ratio. If you want to sacrifice speed for better compression, you can change the config to use zstd as the compression method.

1

sed -i 's/_zram_algorithm=.*/_zram_algorithm="zstd"/g' /etc/default/zram-swap

Apply

1

systemctl restart zram-swap

Tune kernel parameters to make better use of ZRAM

1
2
3
4
5
6


cat <<EOF > /etc/sysctl.d/zram.conf
vm.swappiness = 180
vm.watermark_boost_factor = 0
vm.watermark_scale_factor = 125
vm.page-cluster = 0
EOF

Apply

1

sysctl --system

Enable IPv6 SLAAC

Proxmox disables IPv6 by default. To enable IPv6 SLAAC:

1
2
3
4
5
6
7
8


cat <<EOF > /etc/sysctl.d/ipv6.conf
net.ipv6.conf.default.accept_ra = 2
net.ipv6.conf.all.accept_ra = 2
net.ipv6.conf.default.forwarding = 1
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.proxy_ndp = 1
net.ipv6.conf.all.proxy_ndp = 1
EOF

Apply it

1
2
3


sysctl --system
# Note: the following command may break your internet connection.
systemctl restart networking

TRIM Optimizations

TRIM will help prolong SSD lifespan and maintain performance. Enable it on the LVM thin pool.

1

sed -i 's/.*issue_discards = .*/\tissue_discards = 1/g' /etc/lvm/lvm.conf

Since LXCs are just on the host, it will automatically issue discards if the underlying storage supports it, so no further action is needed. If you are using VM, be sure to use SCSI disks (VirtIO SCSI Single) with discard=on option. I’ve heard that VirtIO Block devices will not work with dicards but I haven’t tested it yet. Correct me if I’m wrong.

Limit Journal Size

1

sed -i 's/.*SystemMaxUse.*/SystemMaxUse=32M/g' /etc/systemd/journald.conf

Apply

1
2


systemctl daemon-reload
systemctl restart systemd-journald

NVIDIA P2P Driver Installation

Instead of regular NVIDIA drivers, we will be installing P2P-enabled drivers to force enable PCIe P2P capabilities on consumer cards (like GeForce RTX 4090). This will bring performance boost (~10%) across multiple scenarios. For details, refer to my blog post: Enabling PCIe P2P on NVIDIA RTX 4090s

Disable IOMMU

Since PCIe P2P in Linux doesn’t work so well with IOMMU-enabled systems (there are many potential issues you may run into), you may as well just disable IOMMU. Note that if you have more than 255 CPU cores, you will only have access to 255 CPUs due to APIC fallback.

To do disable IOMMU, you can either

Disable Intel VT-d (AMD is on by default) in BIOS
Disable intel_iommu or amd_iommu in Linux kernel parameters

To disable intel_iommu or amd_iommu in Linux kernel parameters, use the following command to add intel_iommu=off and amd_iommu=off to your GRUB_CMDLINE_LINUX_DEFAULT. BTW: I also reduce the screen resolution to 1024x768 because it’s a server

1

sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=off amd_iommu=off video=1024x768@60"/g' /etc/default/grub

Reboot your system to see the effect.

1

reboot

Make sure the follow command produces NO entries. If there are output, it means IOMMU is not correctly disabled.

1

ls /sys/class/iommu

Disable ACS

You may need to disable ACS to get PCIe P2P to work, refer to https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html (PCI Access Control Services)

Enable Above 4G Decoding

PCIe P2P will need to access each GPU’s memory, so the PCIe BAR should be large enough to cover GPU’s memory. You must enable Above 4G Decoding in your motherboard settings to make the PCIe BAR large enough to work. Enable it in your motherboard’s BIOS.

Install Official Driver

Before we install the P2P kernel module, we need to install the official drivers.

Since we are installing unofficial P2P kernel modules, there is a limited number of driver version that will work. The version of the P2P kernel modules MUST be the same as the official driver we are going to install.

Refer to https://github.com/tinygrad/open-gpu-kernel-modules to see what versions are available. I also have my patches that works with much newer driver versions (575.57.08) (at the time of writing) https://github.com/charlie0129/open-gpu-kernel-modules .

I will be using version 575.57.08, so we should download the official driver of the same version (575.57.08). You should download the .run file, e.g., NVIDIA-Linux-x86_64-575.57.08.run

1
2
3


# Assume you download the installer as NVIDIA-Linux-x86_64-575.57.08.run
installer="NVIDIA-Linux-x86_64-575.57.08.run"
chmod +x $installer

Skip kernel modules because we will install P2P-patched version later.

1

./$installer --no-kernel-modules

Choose whatever you want, doesn’t matter.

Yes, disable nouveau, we will be using NVIDIA drivers.

Do not abort, just continue. Nouveau will be disabled on next boot.

Just choose the default option for every step that comes later.

Remove the driver installation file because we will not be using it later.

1

rm $installer

You can skip reboot now. We will reboot after we installed the kernel modules.

Install P2P-Enabled Kernel Modules

Make sure the version of the P2P-enabled kernel modules match the version of the driver. I will use veriosn 575.57.08.

1

git clone --depth=1 -b 575.57.08-p2p https://github.com/charlie0129/open-gpu-kernel-modules.git

To build kernel modules, install build dependencies and kernel source. sudo installed because the install script uses sudo but we don’t have it now.

1

apt install sudo build-essential proxmox-headers-$(uname -r)

Build and install the kernel module

1

./install.sh

You can safely ignore the NVIDIA-SMI failure as long as the build succeeds (you should see a DEPMOD /lib/modules/6.14.8-2-pve line) because we haven’t rebooted yet (so NVIDIA-SMI can’t be used).

Remove the source after installing

1
2


cd ..
rm -rf open-gpu-kernel-modules

Reboot your system and you should see nvidia-smi running fine.

To tell if P2P is enabled, we will use a simple method (performance testing will be done later):

1

nvidia-smi -q | grep -i bar -A 3

You should see > 2048 MiB BAR1 Total memory, 32768 MiB in my case.

1
2
3
4


 BAR1 Memory Usage
 Total : 32768 MiB
 Used : 2 MiB
 Free : 32766 MiB

Enable Persistence Mode

Persistence mode will:

Keep the GPU driver running so program can start faster
Lower GPU power mode when it’s idle to save power. For RTX 4090s, it can drop from ~70W to ~10W.
Make sure /dev/nvidia* device nodes are ready. This is useful because we are passing the GPU devices /dev/nvidia* to CTs (LXC Containers) later, it requires the device nodes to be present on start up for auto-started CTs.

So you really should enable it.

To enable persistence mode:

1

/usr/bin/nvidia-smi -pm 1

To make it persistent across reboots, we will use a cron job.

1

crontab -e

Add a line to run nvidia-smi on boot:

1

@reboot /usr/bin/nvidia-smi -pm 1 >/dev/null 2>&1

Why not use nvidia-persistenced systemd service? Because for whatever reason, it does not initialize the device nodes /dev/nvidia* correctly, so auto-started CTs will encounter Cuda failure 'unknown error'.
1
2
3
4
cd /usr/share/doc/NVIDIA_GLX-1.0/samples
tar jxf nvidia-persistenced-init.tar.bz2
cd nvidia-persistenced-init
./install.sh

You should feel nvidia-smi runs much faster than before.

Build CT Templates

We will build a CT Template that has everything a student will need:

Common build dependencies
CUDA
Docker
…

Download the Linux distro you want. I chose Ubuntu 24.04, not because I like Ubuntu (I use Debian), but because most students only know Ubuntu.

Create an unprivileged CT just like you normally would. Just remember to give it a bit more disk space (32GB or more) because we will install CUDA later and CUDA is really large.

Install Docker

After you created the CT, shut it down. Add additional settings to allow Docker to use overlayfs driver, otherwise Docker images will be extremely inefficient in CTs.

1
2
3
4
5
6
7


# This command should be run on the host, not CT.
# Assume 8000 is the ID of your template CT
cat <<EOF >> /etc/pve/local/lxc/8000.conf
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
EOF

All following commands should be run in the CT unless specified otherwise.

Add Docker configuration. Systemd journal is used as log driver to make use of previously configured journal size limit. By writing logs to a centralized location, it’s also useful if you want to write logs to memory to reduce disk writes (e.g. log2ram).

1
2
3
4
5
6
7
8


mkdir -p /etc/docker/
cat <<EOF > /etc/docker/daemon.json
{
 "live-restore": true,
 "experimental": true,
 "log-driver": "journald"
}
EOF

Install Docker

1
2


export DOWNLOAD_URL=https://mirrors.ustc.edu.cn/docker-ce
curl -fsSL https://get.docker.io | sh

Revert Containerd Config

Docker modified containerd config. We will revert it to the default config.

1

containerd config default >/etc/containerd/config.toml

Add GPUs to CT

Choose which GPU you need to passthrough by looking for the index in nvidia-smi, or by ls -l /dev/nvidia*

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# This command should be run on the host, not CT.
# ls -l /dev/nvidia*
crw-rw-rw- 1 root root 195, 0 2025-08-17 01:47:58 /dev/nvidia0
crw-rw-rw- 1 root root 195, 1 2025-08-17 01:48:00 /dev/nvidia1
crw-rw-rw- 1 root root 195, 2 2025-08-17 01:48:01 /dev/nvidia2
crw-rw-rw- 1 root root 195, 3 2025-08-17 01:48:02 /dev/nvidia3
crw-rw-rw- 1 root root 195, 4 2025-08-17 01:48:03 /dev/nvidia4
crw-rw-rw- 1 root root 195, 5 2025-08-17 01:48:05 /dev/nvidia5
crw-rw-rw- 1 root root 195, 6 2025-08-17 01:48:06 /dev/nvidia6
crw-rw-rw- 1 root root 195, 7 2025-08-17 01:48:07 /dev/nvidia7
... (omitted)

For example, if want to add all 8 GPUs (0, 1, 2, 3, 4, 5, 6, 7) to CT, I will need to run:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# This command should be run on the host, not CT.
# Assume 8000 is the ID of your template CT
cat <<EOF >> /etc/pve/local/lxc/8000.conf
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia1 dev/nvidia1 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia2 dev/nvidia2 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia3 dev/nvidia3 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia4 dev/nvidia4 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia5 dev/nvidia5 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia6 dev/nvidia6 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia7 dev/nvidia7 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/fb0 dev/fb0 none bind,optional,create=file
EOF

Note the lxc.mount.entry: /dev/nvidiaX dev/nvidiaX none bind,optional,create=file lines. Each line represents a single GPU to add to the CT. I will add 8 GPUs, so I add 8 lines from nvidia0 through nvidia7. Remove or add them as needed.

Check if the GPUs are there inside CT:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# ls -l /dev/nvidia*
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:00 /dev/nvidia-modeset
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:50:43 /dev/nvidia-uvm
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:50:43 /dev/nvidia-uvm-tools
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:47:58 /dev/nvidia0
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:00 /dev/nvidia1
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:01 /dev/nvidia2
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:02 /dev/nvidia3
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:03 /dev/nvidia4
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:05 /dev/nvidia5
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:06 /dev/nvidia6
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:07 /dev/nvidia7
crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:47:58 /dev/nvidiactl

We haven’t installed drivers yet, so nvidia-smi is not available.

Install GPU Driver

Important: you should use the same driver version as the host (575.57.08 in my case).

1
2
3
4


wget <driver-runfile-url> -O driver.run
chmod +x driver.run
driver.run --no-kernel-modules
rm driver.run

Just install as normal.

Now you should be able to use nvidia-smi and see the GPUs you added to the CT (8 GPUs in my case).

Install CUDA

Important: current driver version MUST be greater than (or equal to) the driver required by CUDA. For example, CUDA 12.9.0 will work on my machine because CUDA 12.9.0 requires 575.00, and my current driver version is 575.57.08 (greater than 575.00).

1
2
3
4


wget <cuda-runfile-url> -O cuda.run
chmod +x cuda.run
cuda.run
rm cuda.run

Remember to deselect the driver (we already installed the correct driver):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


┌──────────────────────────────────────────────────────────────────────────────┐
│ CUDA Installer │
│ - [ ] Driver <== deselect this │
│ [ ] 575.51.03 <== deselect this │
│ + [X] CUDA Toolkit 12.9 │
│ [ ] CUDA Demo Suite 12.9 <== deselect this (not useful, takes up space) │
│ [ ] CUDA Documentation 12.9 <== deselect this (not useful, takes up space) │
│ - [ ] Kernel Objects │
│ [ ] nvidia-fs │
│ Options │
│ Install │
│ │
│ Up/Down: Move | Left/Right: Expand | 'Enter': Select | 'A': Advanced options │
└──────────────────────────────────────────────────────────────────────────────┘

After CUDA installation, configure PATH and LD_LIBRARY_PATH per printed instructions.

1
2
3
4
5


... (omitted)
Please make sure that
 - PATH includes /usr/local/cuda-12.9/bin
 - LD_LIBRARY_PATH includes /usr/local/cuda-12.9/lib64, or, add /usr/local/cuda-12.9/lib64 to /etc/ld.so.conf and run ldconfig as root
... (omitted)

Note that you can use /usr/local/cuda instead of /usr/local/cuda-XX.X so it’s independent of CUDA versions.

Also add export CUDA_HOME=/usr/local/cuda to your shell rc file.

nvcc should be available after a shell reload:

1
2
3
4
5
6


# nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Apr__9_19:24:57_PDT_2025
Cuda compilation tools, release 12.9, V12.9.41
Build cuda_12.9.r12.9/compiler.35813241_0

Install NCCL

You will need NCCL for most GPU-related communications.

You should find a NCCL that’s compatible with your CUDA version. For example, NCCL 2.27.3 is compatible with my CUDA version 12.9.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


# Download NCCL package
wget https://xxx/nccl_2.27.3-1+cuda12.9_x86_64.txz

tar Jxf nccl_2.27.3-1+cuda12.9_x86_64.txz

# Set some variables
NCCL_VERSION=2.27

mkdir -p /usr/local/nccl-$NCCL_VERSION

cp -vRf nccl_2.27.3-1+cuda12.9_x86_64/* /usr/local/nccl-$NCCL_VERSION
rm -rf nccl_2.27.3-1+cuda12.9_x86_64

ln -s /usr/local/nccl-$NCCL_VERSION /usr/local/nccl

echo "/usr/local/nccl/lib" >>/etc/ld.so.conf
ldconfig

ln -s /usr/local/nccl/include/nccl.h /usr/local/include/nccl.h

# Remove NCCL package
rm https://xxx/nccl_2.27.3-1+cuda12.9_x86_64.txz

Also add export NCCL_HOME=/usr/local/nccl in your shell rc file,

Install NVIDIA Container Toolkit

To run Docker containers with GPUs.

Note that the URLs below are mirrors in China. You can use original URL if you prefer.

1
2
3
4
5
6
7
8


curl -fsSL https://mirrors.ustc.edu.cn/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://mirrors.ustc.edu.cn/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
 sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
 tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sed -i 's/nvidia.github.io/mirrors.ustc.edu.cn/g' /etc/apt/sources.list.d/nvidia-container-toolkit.list

apt-get update
apt-get install -y nvidia-container-toolkit

Configure Docker and Containerd to use NVIDIA Container Toolkit

1
2


nvidia-ctk runtime configure --runtime=docker
nvidia-ctk runtime configure --runtime=containerd

Since we are running inside unprivileged containers, we need to set no-cgroups to true

1

nvidia-ctk config --set nvidia-container-cli.no-cgroups --in-place

Install NVIDIA Nsight Systems

So we can profile applications running on the GPU.

1
2
3


wget https://xxx/NsightSystems-linux-cli-public-2025.3.1.90-3582212.deb
dpkg -i NsightSystems-linux-cli-public-2025.3.1.90-3582212.deb
rm -f NsightSystems-linux-cli-public-2025.3.1.90-3582212.deb

Functionality Tests

NCCL

1
2
3
4


git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
git reset --hard 903918f # I only tested this commit.
make

P2P ON: NCCL_P2P_LEVEL=sys ./build/all_reduce_perf --minbytes 8 --maxbytes 128M --stepfactor 2 --ngpus 8 (change --ngpus accordingly)

P2P OFF: NCCL_P2P_DISABLE=1 ./build/all_reduce_perf --minbytes 8 --maxbytes 128M --stepfactor 2 --ngpus 8 (change --ngpus accordingly)

You should see a bandwidth increase after P2P is on.

CUDA P2P

1
2
3
4
5


git clone https://github.com/NVIDIA/cuda-samples.git
git reset --hard 9c688d7 # I only tested this commit.
cd cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest
make
./p2pBandwidthLatencyTest

You should see lower GPU-GPU latency, higher bandwidth if P2P is on.

Prepare for Templating

Check if there are unnecessary files and remove them.

1
2


cd
ls -l

Remove history

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


cd

unset HISTFILE
rm .*_history

echo -n >/var/log/lastlog
echo -n >/var/log/wtmp
echo -n >/var/log/btmp

journalctl --vacuum-time=1s

rm -f /var/log/*.log

Now, it’s ready for converting to a template. To give each student a new development container, just clone this template.

Other Minor Settings

RAID Card Write Back

To achieve maximum write speed, you can enable write back mode on your RAID card, so data is written to cache first. This will significantly (yes, by A LOT, sometimes over 100x) improve random write speed on HDDs (SSDs are already very fast). If your RAID card has a battery, you don’t need to worry about data integrity in case a power failure.

Write cache policy:

Write Throuh: no cache
Write Back: cached if battery backup is present on the RAID card
Always Write Back: always cached

Obstacles

Only 255 Cores are Recognized - x2APIC

I noticed that I have one CPU offline. It should recognize 256 CPUs but I only have 255 CPUs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# lscpu
Architecture: x86_64
 CPU op-mode(s): 32-bit, 64-bit
 Address sizes: 48 bits physical, 48 bits virtual
 Byte Order: Little Endian
CPU(s): 256
 On-line CPU(s) list: 0-254
 Off-line CPU(s) list: 255 <--------------- one CPU is offline -----------
Vendor ID: AuthenticAMD
 Model name: AMD EPYC 7763 64-Core Processor
 CPU family: 25
 Model: 1
 Thread(s) per core: 2
 Core(s) per socket: 64
 Socket(s): 2

After some digging, I learned that I need to enable x2APIC on both the motherboard and kernel to have more than 255 CPU cores recognized.

In BIOS Local APIC Mode should already be x2APIC .

Now I realized it can be because I disabled IOMMU for PCIe P2P before. It turns out that disableing IOMMU can limit the number of available logical cores to 255. The reason is that the Linux kernel disables x2APIC in this case and falls back to APIC, which can only enumerate a maximum of 255 (logical) cores.

I will keep IOMMU disabled because I need PCIe P2P to function (I don’t want to deal with P2P with IOMMU on). Just forget about the 256th core :p.

(Chinese-Only) 一个Linux用户的绿联NAS配置指南

Sun, 27 Jul 2025 16:45:00 +0800

启用 SSH

没SSH就别想着用Terminal了，赶紧去开了。

去绿联NAS的Web管理界面，控制面板 -> 终端机 -> SSH ，记得关闭自动关闭功能。

配置 root 用户 SSH 登录

用你的NAS用户名SSH到NAS上ssh yourname@nas-ip

你需要先有一个SSH密钥对，如果你还没有，可以用 ssh-keygen -t rsa 生成一个。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# 切root
sudo su

# 打开 root SSH 登录（仅允许密钥登录）
# 注意，别去改 /etc/ssh/sshd_config ，因为绿联NAS的系统会保护这个文件，改了重启就失效了。
echo 'PermitRootLogin prohibit-password' >> /etc/ssh/sshd_config.d/A0-root-login.conf

# 应用修改
systemctl restart ssh

# 将你的公钥添加到 /root/.ssh/authorized_keys 中
mkdir -p /root/.ssh
chmod 700 /root/.ssh
echo 'your-public-key' >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys

配置你本机的 ~/.ssh/config 文件使用对应的密钥登录NAS：

1
2
3
4


Host nas
 HostName nas-ip
 User root
 IdentityFile ~/.ssh/<private-key>

安装包管理器 Opkg

那么有人会问，你为什么要装包管理器，你怎么不用自带的 apt？UGOS Pro不是基于Debian的么？其实最主要的原因是避免干扰内置预装的package。我举个例子，你想安装git，好，那你得apt update吧，update完拉过来了最新的package index，然后刚好这个新的git依赖新版的libcurl，你装的时候给他一起升级了，但是绿联自带了不少预装的package，这些package可能是根据老版本libcurl编译的。正常的debian你upgrade一下就完事了，但是UGOS作为一个定制的Debian，不建议你这么做（其实要是你尝试去upgrade，你会发现不让你upgrade，绿联已经做了保护了，虽然你可以绕过就是）。

OK，我们解释清楚了为什么不用apt。那么用什么呢？我的建议是一个轻量级给嵌入式设备用的包管理器，用OpenWRT的可能很熟悉了，就是opkg。你想重量级一点，也可以用Homebrew/Linuxbrew（不过你得先装git才能装Homebrew，但是git又没有，现在还没包管理器也没得装git，循环依赖了属于是），也可以用Nix（我觉得Nix太重了，除非你真的需要Nix的特性，不然就算了）。

这里用 Opkg 做演示。你需要用root用户SSH到NAS上ssh root@nas-ip。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# 下载 opkg 安装脚本
wget https://mirrors.nju.edu.cn/entware/x64-k3.2/installer/generic.sh -O opkg-install.sh
# 换源 1
sed -i 's|http://bin.entware.net|https://mirrors.nju.edu.cn/entware|g' opkg-install.sh
# 换源 2
sed -i 's|-O /opt/etc/opkg.conf|-O /opt/etc/opkg.conf \&\& sed -i "s,http://bin.entware.net,https://mirrors.nju.edu.cn/entware,g" /opt/etc/opkg.conf|g' opkg-install.sh
# 执行 opkg 安装脚本
/bin/sh opkg-install.sh

# 临时设置 PATH ，使得 opkg 命令可用
export PATH=/opt/bin:/opt/sbin:/opt/usr/bin:$PATH
# 设置 PATH ，使得 opkg 命令可用。如果你用 zsh，记得改成 ~/.zshrc 。
# 或者如果你用我的 dotfiles，现在可以先不做这一步，你可以看后续我配置 zsh 的时候怎么做的。
echo 'export PATH=/opt/bin:/opt/sbin:/opt/usr/bin:$PATH' >> ~/.bashrc

安装常用Package

这时有包管理器了，而且是与NAS系统隔离的，你可以随意安装了。以下是一些常用的包：

1
2
3
4
5
6


# 注意 vim 需要安装两个包，vim-full 和 vim-runtime
opkg install vim-full vim-runtime
# 终于能装 git 了，注意需要 git-http
opkg install git git-http
# 其他常见工具
opkg install htop sysstat zsh

配置 ZSH

如果你想要一个更好的终端体验，建议使用 ZSH 和比较好的配置。以下是使用我的 dotfiles 来配置 ZSH 的步骤：

注意，你需要在你以后会经常使用的用户中配置 ZSH 。我这里都是用的 root 用户，我对 Linux 比较熟悉了所以我并不怕搞坏东西。如果你不熟悉 Linux，建议切换回自己用户配置 ZSH 。如果找不到 git ，记得参考 opkg 安装中最后一步重新设置 PATH 。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# 先设置代理，等会要从 GitHub 上下载文件
export https_proxy=http://<your-proxy-ip>:<your-proxy-port>
# 克隆我的 dotfiles 仓库
cd
git clone https://github.com/charlie0129/dotfiles.git
cd dotfiles
./bootstrap.sh -f
zsh
# 询问 Do you need to use a proxy [y/n] 的时候选择 n （因为前面 set 过 proxy 了）
# 询问 Change login shell of root to /opt/bin/zsh 的时候选择 y （不然呢）

将 Opkg 的 bin 和 sbin 目录添加到 ZSH 的 PATH 中：

1

vim ~/dotfiles/env/custom.sh

1
2
3
4
5
6
7
8
9


# This list is inserted before PATH
PATH_BEFORE=(
 # custom bin in this repo, i.e. bin/custom
 $HOME/dotfiles/bin/custom
 # Opkg bin and sbin directories
 /opt/bin
 /opt/sbin
 /opt/usr/bin
)

效果

优化 Zram

需要使用 root 用户

绿联默认的 zram 不够激进，关了它。我们要用 zstd 算法和一半的内存来做zram，榨干内存。

1
2
3
4
5
6
7


# 先设置代理，等会要从 GitHub 上下载文件
export https_proxy=http://<your-proxy-ip>:<your-proxy-port>
git clone --depth=1 https://github.com/foundObjects/zram-swap.git
cd zram-swap
./install.sh
cd ..
rm -rf zram-swap

配置使用 zstd 算法（因为 Debian 13 里面默认 zram 不给 lzo-rle 算法了，lz4 的压缩率又不行，所以用 zstd ）：

1
2


sed -i 's/_zram_algorithm=.*/_zram_algorithm="zstd"/g' /etc/default/zram-swap
systemctl restart zram-swap

为 zram 配置合适的 sysctl ，较大的 swappiness 值可以不活跃的页面更快地被交换出去，让内存留给更有用的页面。因为 sysctl.conf 里面绿联已经设置了 vm.swappiness 如果放在 /etc/sysctl.d/ 里面会被覆盖掉，所以直接放在 sysctl.conf 里面。

1
2
3
4
5
6
7


echo '# ZRAM BEGIN' >> /etc/sysctl.conf
echo 'vm.swappiness = 180' >> /etc/sysctl.conf
echo 'vm.watermark_boost_factor = 0' >> /etc/sysctl.conf
echo 'vm.watermark_scale_factor = 125' >> /etc/sysctl.conf
echo 'vm.page-cluster = 0' >> /etc/sysctl.conf
echo '# ZRAM END' >> /etc/sysctl.conf
sysctl --system

能看到内存大小 1.5 倍的zram swap即成功

1
2
3


$ swapon
NAME TYPE SIZE USED PRIO
/dev/zram0 partition 93.8G 0B 15

安装 Docker

先去应用中心安装 Docker 。默认绿联的Docker 配置比较烂，而且还会在根目录下留下一个 /daemon.json 的文件夹（WTF？），一看就是绿联的安装脚本写错了。

1
2


# 删除错误的 daemon.json 文件夹
rmdir /daemon.json

将以下json写入 /etc/docker/daemon.json：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


{
 "log-opts": {
 "max-size": "1m"
 },
 "experimental": true,
 "metrics-addr": "0.0.0.0:8132",
 "data-root": "<your-data-root>",
 "registry-mirrors": [
 "https://xxxxxx.com/dockerhub/",
 ]
}

log-opts：限制日志大小，防止日志占满磁盘。
experimental：启用实验性功能。
metrics-addr：设置可观测端口。
data-root：设置 Docker 数据目录，建议查看你之前的 /etc/docker/daemon.json 中的配置，不要动。
registry-mirrors：设置 Docker 镜像加速器，建议使用国内的镜像源。

可观测配置

Grafana: 用于可视化和监控。
Victoria Metrics: 用于指标存储与查询。
Alloy: 用于容器日志采集。
Cadvisor: 用于容器指标采集。
Loki: 用于日志存储与查询。
Node Exporter: 用于主机指标采集。
Smartctl Exporter: 用于硬盘SMART指标采集。
Intel PCM: 用于CPU性能监控。

效果：TODO

参考 Docker Compose （镜像版本可以适当升级，记得更换volume中数据存储位置）：

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173


name: o11y
services:
 grafana:
 container_name: grafana
 image: grafana/grafana:12.1.0
 restart: unless-stopped
 mem_limit: 1G
 user: 0:0
 networks:
 - o11y
 # 我使用了 Traefik 作为反向代理，所以不需要暴露端口
 # 如果你没有使用 Traefik，可以取消注释以下端口映射
 # ports:
 # - 3000:3000
 healthcheck:
 test: wget --no-verbose --tries=1 --spider http://localhost:3000/api/health
 volumes:
 - /volume2/docker/data/grafana:/var/lib/grafana
 environment:
 TZ: Asia/Shanghai
 GF_SERVER_ENABLE_GZIP: true
 # 如果你使用 Traefik 作为反向代理，可以添加以下标签
 labels:
 - "traefik.enable=true"
 - "traefik.http.routers.grafana.entrypoints=web"
 - "traefik.http.routers.grafana.rule=Host(`grafana.example.com`)"
 - "traefik.http.routers.grafana.service=grafana-secure"
 # - "traefik.http.routers.grafana.middlewares=grafana-https-redirect"
 # - "traefik.http.middlewares.grafana-https-redirect.redirectscheme.scheme=https"
 - "traefik.http.routers.grafana-secure.tls=true"
 - "traefik.http.routers.grafana-secure.tls.certresolver=cloudflare"
 - "traefik.http.routers.grafana-secure.entrypoints=websecure"
 - "traefik.http.routers.grafana-secure.rule=Host(`grafana.example.com`)"
 - "traefik.http.services.grafana-secure.loadbalancer.server.port=3000"

 vm:
 container_name: vm
 image: victoriametrics/victoria-metrics:v1.122.0
 restart: unless-stopped
 mem_limit: 4G
 user: 0:0
 networks:
 - o11y
 # ports:
 # - 8428:8428
 healthcheck:
 test: wget --no-verbose --tries=1 --spider http://localhost:8428/ || exit 1
 command:
 - -httpListenAddr=0.0.0.0:8428
 - -promscrape.config=/etc/victoriametrics/scrape.yml
 - -storageDataPath=/var/victoriametrics
 - -retentionPeriod=20y
 - -inmemoryDataFlushInterval=300s # 减少写盘
 extra_hosts:
 - 'host.docker.internal:host-gateway'
 - 'd48t:host-gateway'
 volumes:
 - ./vm:/etc/victoriametrics # Scrape 配置
 - /volume2/docker/data/vm:/var/victoriametrics

 alloy:
 container_name: alloy
 image: grafana/alloy:v1.10.0
 restart: unless-stopped
 user: 0:0
 networks:
 - o11y
 mem_limit: 512M
 # ports:
 # - 12345:12345
 command:
 - run
 - --disable-reporting
 - --server.http.listen-addr=0.0.0.0:12345
 - --storage.path=/var/lib/alloy/data
 - /etc/alloy/config.alloy
 volumes:
 - /var/run/docker.sock:/var/run/docker.sock:ro
 - /tmp/alloy:/var/lib/alloy/data # 这玩意巨能写，SSD会写爆，给他放内存里去。绿联的 /tmp 是内存盘。
 - ./alloy/config.alloy:/etc/alloy/config.alloy

 cadvisor:
 container_name: cadvisor
 image: gcr.io/cadvisor/cadvisor:v0.53.0
 mem_limit: 256M
 restart: unless-stopped
 user: 0:0
 # ports:
 # - 8080:8080
 networks:
 - o11y
 volumes:
 - /:/rootfs:ro
 - /var/run:/var/run:ro
 - /sys:/sys:ro
 - /var/lib/docker/:/var/lib/docker:ro
 - /dev/disk/:/dev/disk:ro
 privileged: true
 devices:
 - /dev/kmsg
 command:
 - --store_container_labels=false
 - --docker_only=true
 - --housekeeping_interval=30s

 loki:
 container_name: loki
 image: grafana/loki:3.5.3
 restart: unless-stopped
 user: 0:0
 networks:
 o11y:
 ipv4_address: "172.26.195.254" # 固定 IP，方便 Docker 的 Loki plugin 用（虽然我们这里用的 Alloy 其实并不需要）
 # 如果你没有使用 Traefik 作为反向代理，可以取消注释以下端口映射。
 # ports:
 # - 3100:3100
 mem_limit: 4G
 command:
 - "-config.file=/etc/loki/config.yml"
 volumes:
 - ./loki:/etc/loki:ro
 - /volume2/docker/data/loki:/loki

 node-exporter:
 container_name: node-exporter
 image: prom/node-exporter:v1.9.1
 restart: unless-stopped
 mem_limit: 256M
 user: 0:0
 privileged: true
 # ports:
 # - 9100:9100
 command:
 - --path.rootfs=/host
 - --web.listen-address=172.17.0.1:9100
 network_mode: host
 # networks:
 # - o11y
 pid: host
 volumes:
 - '/:/host:ro,rslave'

 smartctl-exporter:
 container_name: smartctl-exporter
 image: prometheuscommunity/smartctl-exporter:v0.14.0
 restart: unless-stopped
 mem_limit: 512M
 user: 0:0
 # ports:
 # - 192.168.91.1:9633:9633
 networks:
 - o11y
 privileged: true

 pcm:
 container_name: pcm
 image: opcm/pcm
 mem_limit: 256M
 restart: unless-stopped
 user: 0:0
 privileged: true
 # ports:
 # - 192.168.91.1:9738:9738
 networks:
 - o11y

networks:
 o11y:
 name: o11y
 driver: bridge
 ipam:
 config:
 - subnet: "172.26.195.0/24"

参考 Victoria Metrics 的 Scrape 配置：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


# ./vm/scrape.yml
global:
 scrape_interval: 15s

scrape_configs:
 - job_name: victoria-metrics
 static_configs:
 - targets:
 - vm:8428
# 如果你有 Traefik
# - job_name: traefik
# static_configs:
# - targets:
# - traefik:8082
 - job_name: node-exporter
 static_configs:
 - targets:
 - d48t:9100
 - job_name: cadvisor
 static_configs:
 - targets:
 - cadvisor:8080
 - job_name: loki
 static_configs:
 - targets:
 - loki:3100
 - job_name: grafana
 static_configs:
 - targets:
 - grafana:3000
 - job_name: smartctl-exporter
 static_configs:
 - targets:
 - smartctl-exporter:9633
 - job_name: pcm
 static_configs:
 - targets:
 - pcm:9738

参考 Loki 配置：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


# ./loki/config.yml
auth_enabled: false

server:
 http_listen_port: 3100

common:
 instance_addr: 127.0.0.1
 path_prefix: /loki
 storage:
 filesystem:
 chunks_directory: /loki/chunks
 rules_directory: /loki/rules
 replication_factor: 1
 ring:
 kvstore:
 store: inmemory

ingester:
 chunk_encoding: lz4 # 比默认的snappy压缩率高
 chunk_target_size: 8388608 # 8M
 max_chunk_age: 48h
 chunk_idle_period: 12h

limits_config:
 max_query_lookback: 672h # 28 days
 retention_period: 672h  # 28 days

schema_config:
 configs:
 - from: 2020-10-24
 store: tsdb
 object_store: filesystem
 schema: v13
 index:
 prefix: index_
 period: 24h

ruler:
 alertmanager_url: http://localhost:9093

参考 Grafana Alloy 配置：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57


# ./alloy/config.alloy
logging {
 level = "info"
 format = "logfmt"
}

// Discover Docker containers and extract metadata.
discovery.docker "logs_integrations_docker" {
 host = "unix:///var/run/docker.sock"
 refresh_interval = "5s"
}

// Define a relabeling rule to create a service name from the container name.
discovery.relabel "logs_integrations_docker" {
 targets = []


 rule {
 target_label = "job"
 replacement = "integrations/docker"
 }


 rule {
 target_label = "instance"
 replacement = "d48t" // constants.hostname
 }


 rule {
 source_labels = ["__meta_docker_container_name"]
 regex = "/(.*)"
 target_label = "container"
 }


 rule {
 source_labels = ["__meta_docker_container_log_stream"]
 target_label = "stream"
 }
}


// Configure a loki.source.docker component to collect logs from Docker containers.
loki.source.docker "logs_integrations_docker" {
 host = "unix:///var/run/docker.sock"
 targets = discovery.docker.logs_integrations_docker.targets
 relabel_rules = discovery.relabel.logs_integrations_docker.rules
 forward_to = [loki.write.local.receiver]
 refresh_interval = "15s"
}

loki.write "local" {
 endpoint {
 url = "http://loki:3100/loki/api/v1/push"
 }
}

RTX4090 GPUDirect P2P Unlocked!

Fri, 07 Feb 2025 11:18:00 +0800

If you want see the test results directly, please jump to the Test the unlocked performance section.

What is P2P?

From NVIDIA: GPUDirect Peer to Peer enables GPU-to-GPU copies as well as loads and stores directly over the memory fabric (PCIe, NVLink). GPUDirect Peer to Peer is supported natively by the CUDA Driver.

As you can see, GPUDirect P2P allows for direct memory access between GPUs, bypassing the CPU and system memory. This can lead to significant performance improvements in multi-GPU systems. For example, in a multi-GPU LLM inference system, the data can be directly transferred between GPUs without involving the CPU, reducing latency and improving throughput.

Why is P2P disabled on RTX4090?

Easy, NVIDIA wants to make money. They want you to buy the more expensive Tesla GPUs if you want P2P.

Test machine configuration

CPU: 2 x AMD EPYC 7542 (32C 64T, 225W TDP, 2.9GHz base, 3.4GHz boost)
MEM: 2 x 8 x 32GB 3200MT/s DDR4 ECC
GPU: 8 x NVIDIA GeForce RTX 4090 (24GB GDDR6X, 450W TDP)

Topology (ideally, the GPUs should be behind a PCIe switch, but our test machine doesn’t have one):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


 GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS 24-31,88-95 3 N/A
GPU1 SYS X SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS 16-23,80-87 2 N/A
GPU2 SYS SYS X SYS SYS SYS SYS SYS SYS SYS SYS SYS 8-15,72-79 1 N/A
GPU3 SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS SYS 0-7,64-71 0 N/A
GPU4 SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS 56-63,120-127 7 N/A
GPU5 SYS SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS 48-55,112-119 6 N/A
GPU6 SYS SYS SYS SYS SYS SYS X SYS PHB PHB PHB PHB 40-47,104-111 5 N/A
GPU7 SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS 32-39,96-103 4 N/A
NIC0 SYS SYS SYS SYS SYS SYS PHB SYS X PIX PHB PHB
NIC1 SYS SYS SYS SYS SYS SYS PHB SYS PIX X PHB PHB
NIC2 SYS SYS SYS SYS SYS SYS PHB SYS PHB PHB X PIX
NIC3 SYS SYS SYS SYS SYS SYS PHB SYS PHB PHB PIX X

Legend:

 X = Self
 SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
 NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
 PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
 PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
 PIX = Connection traversing at most a single PCIe bridge
 NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

 NIC0: mlx5_0
 NIC1: mlx5_1
 NIC2: mlx5_2
 NIC3: mlx5_3

Test the original performance

PCI BAR

Only 256MB of BAR memory is available. Note that I did enabled Resizable BAR in the BIOS (not my fault).

1
2
3
4
5


# nvidia-smi -q | grep -i bar -A 3
 BAR1 Memory Usage
 Total : 256 MiB
 Used : 1 MiB
 Free : 255 MiB

NCCL Tests

Building NCCL Tests

1
2
3


git clone --depth=1 https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
make -j$(nproc)

Runnning:

Around 14.5GB/s of bandwidth.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


# ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 8
# nThread 1 nGpus 8 minBytes 8 maxBytes 134217728 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
# Rank 0 Group 0 Pid 1756513 on <REDACTED> device 0 [0x01] NVIDIA GeForce RTX 4090
# Rank 1 Group 0 Pid 1756513 on <REDACTED> device 1 [0x25] NVIDIA GeForce RTX 4090
# Rank 2 Group 0 Pid 1756513 on <REDACTED> device 2 [0x41] NVIDIA GeForce RTX 4090
# Rank 3 Group 0 Pid 1756513 on <REDACTED> device 3 [0x61] NVIDIA GeForce RTX 4090
# Rank 4 Group 0 Pid 1756513 on <REDACTED> device 4 [0x81] NVIDIA GeForce RTX 4090
# Rank 5 Group 0 Pid 1756513 on <REDACTED> device 5 [0xa1] NVIDIA GeForce RTX 4090
# Rank 6 Group 0 Pid 1756513 on <REDACTED> device 6 [0xc1] NVIDIA GeForce RTX 4090
# Rank 7 Group 0 Pid 1756513 on <REDACTED> device 7 [0xe1] NVIDIA GeForce RTX 4090
#
# out-of-place in-place 
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
 8 2 float sum -1 38.83 0.00 0.00 0 40.43 0.00 0.00 0
 16 4 float sum -1 38.73 0.00 0.00 0 38.59 0.00 0.00 0
 32 8 float sum -1 41.85 0.00 0.00 0 42.55 0.00 0.00 0
 64 16 float sum -1 42.55 0.00 0.00 0 42.65 0.00 0.00 0
 128 32 float sum -1 42.42 0.00 0.01 0 42.88 0.00 0.01 0
 256 64 float sum -1 42.44 0.01 0.01 0 42.53 0.01 0.01 0
 512 128 float sum -1 42.58 0.01 0.02 0 42.68 0.01 0.02 0
 1024 256 float sum -1 44.04 0.02 0.04 0 42.39 0.02 0.04 0
 2048 512 float sum -1 42.91 0.05 0.08 0 42.61 0.05 0.08 0
 4096 1024 float sum -1 43.18 0.09 0.17 0 42.69 0.10 0.17 0
 8192 2048 float sum -1 44.02 0.19 0.33 0 42.96 0.19 0.33 0
 16384 4096 float sum -1 43.10 0.38 0.67 0 42.99 0.38 0.67 0
 32768 8192 float sum -1 44.18 0.74 1.30 0 43.63 0.75 1.31 0
 65536 16384 float sum -1 45.66 1.44 2.51 0 45.28 1.45 2.53 0
 131072 32768 float sum -1 67.27 1.95 3.41 0 54.86 2.39 4.18 0
 262144 65536 float sum -1 124.8 2.10 3.68 0 121.1 2.17 3.79 0
 524288 131072 float sum -1 196.9 2.66 4.66 0 200.5 2.61 4.58 0
 1048576 262144 float sum -1 222.8 4.71 8.24 0 226.1 4.64 8.12 0
 2097152 524288 float sum -1 373.4 5.62 9.83 0 358.6 5.85 10.23 0
 4194304 1048576 float sum -1 580.8 7.22 12.64 0 592.0 7.09 12.40 0
 8388608 2097152 float sum -1 1041.8 8.05 14.09 0 1040.2 8.06 14.11 0
 16777216 4194304 float sum -1 1972.8 8.50 14.88 0 1979.5 8.48 14.83 0
 33554432 8388608 float sum -1 3905.6 8.59 15.04 0 3919.0 8.56 14.98 0
 67108864 16777216 float sum -1 7907.1 8.49 14.85 0 7905.3 8.49 14.86 0
 134217728 33554432 float sum -1 16253 8.26 14.45 0 16227 8.27 14.47 0
# Out of bounds values : 0 OK
# Avg bus bandwidth : 4.85265
#

P2P Bandwidth and Latency

Building:

1
2
3


git clone --depth=1 https://github.com/NVIDIA/cuda-samples.git
cd cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest
make -j$(nproc)

Running:

About 21GB/s bidirectional bandwidth
10us+ GPU-GPU latency

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91


# ./p2pBandwidthLatencyTest
P2P Connectivity Matrix
 D\D 0 1 2 3 4 5 6 7
 0 1 0 0 0 0 0 0 0
 1 0 1 0 0 0 0 0 0
 2 0 0 1 0 0 0 0 0
 3 0 0 0 1 0 0 0 0
 4 0 0 0 0 1 0 0 0
 5 0 0 0 0 0 1 0 0
 6 0 0 0 0 0 0 1 0
 7 0 0 0 0 0 0 0 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
 D\D 0 1 2 3 4 5 6 7
 0 909.49 18.77 19.56 20.23 18.72 19.29 19.34 19.83
 1 19.95 919.12 18.93 20.18 18.83 19.23 19.24 19.80
 2 19.87 19.55 919.66 20.27 18.82 19.28 19.29 19.77
 3 19.92 19.48 19.56 918.58 18.82 19.23 19.19 19.80
 4 19.88 20.58 19.74 20.54 920.15 18.38 18.89 18.71
 5 20.04 20.19 19.80 20.37 18.75 921.29 18.88 19.08
 6 20.04 20.11 19.71 20.60 18.77 18.58 920.74 18.62
 7 20.02 20.21 19.73 20.33 18.37 18.74 18.89 921.29
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
 D\D 0 1 2 3 4 5 6 7
 0 910.02 18.83 19.55 20.27 18.76 19.36 19.26 19.83
 1 19.95 920.20 18.92 20.19 18.83 19.26 19.21 19.80
 2 19.82 19.61 919.66 20.28 18.86 19.29 19.34 19.75
 3 19.91 19.47 19.60 920.20 18.83 19.26 19.16 19.81
 4 19.78 20.43 19.75 20.56 920.25 18.60 18.85 18.75
 5 20.02 20.24 19.76 20.45 18.80 922.37 18.89 19.15
 6 20.06 20.12 19.75 20.59 18.80 18.58 921.25 18.61
 7 20.02 20.16 19.80 20.38 18.41 18.72 18.87 920.83
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
 D\D 0 1 2 3 4 5 6 7
 0 916.15 21.03 21.53 21.54 20.69 21.02 21.10 21.06
 1 21.39 920.74 21.59 21.52 20.55 21.02 20.50 21.18
 2 21.15 21.48 921.83 21.20 20.58 21.11 20.98 21.09
 3 21.38 21.67 21.42 922.65 20.53 21.05 20.65 21.21
 4 20.86 20.89 20.86 20.95 923.69 19.99 20.11 19.97
 5 20.90 20.97 21.00 21.02 20.22 922.10 20.15 20.30
 6 20.94 20.62 20.89 20.61 19.98 20.07 923.11 19.99
 7 21.07 21.13 21.10 21.22 20.00 20.29 20.11 923.46
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
 D\D 0 1 2 3 4 5 6 7
 0 917.23 20.86 21.49 21.56 20.63 20.99 21.05 21.13
 1 21.30 922.37 21.61 21.56 20.53 21.08 20.49 21.15
 2 21.15 21.47 922.36 21.16 20.50 21.06 20.94 21.10
 3 21.39 21.60 21.39 922.92 20.63 21.09 20.69 21.22
 4 20.89 20.86 20.81 20.89 923.46 20.12 20.05 20.02
 5 20.88 20.98 20.99 20.87 20.20 923.65 20.12 20.39
 6 21.05 20.63 20.89 20.67 20.07 20.05 922.92 20.02
 7 21.01 21.12 21.06 21.19 20.01 20.31 20.13 923.16
P2P=Disabled Latency Matrix (us)
 GPU 0 1 2 3 4 5 6 7
 0 1.54 20.26 11.68 11.60 12.65 15.80 12.19 11.75
 1 11.60 1.48 11.58 11.66 16.03 12.13 12.55 14.59
 2 20.27 11.59 1.43 20.08 13.17 17.14 16.31 11.41
 3 11.50 11.59 11.51 1.47 13.11 16.14 14.04 11.61
 4 12.53 14.27 12.75 12.65 1.51 12.47 12.49 12.41
 5 11.88 11.73 11.65 11.63 12.28 1.42 14.95 15.40
 6 12.27 12.63 12.42 12.68 12.25 12.30 1.47 11.89
 7 11.46 16.67 11.44 11.47 12.30 13.33 13.57 1.41

 CPU 0 1 2 3 4 5 6 7
 0 4.04 12.23 10.01 9.91 10.88 10.78 10.60 11.29
 1 9.79 3.15 9.85 10.06 10.77 10.56 10.50 10.48
 2 10.04 9.63 3.24 9.94 10.86 10.65 10.70 10.63
 3 9.75 9.61 9.85 3.26 10.92 10.82 10.77 10.67
 4 10.50 10.26 10.61 10.56 3.57 11.55 11.48 11.26
 5 10.52 10.07 10.40 10.48 11.42 3.52 11.32 11.24
 6 10.24 10.20 10.44 10.47 11.42 11.29 3.54 11.35
 7 10.26 10.09 10.50 10.41 11.47 11.31 11.36 3.53
P2P=Enabled Latency (P2P Writes) Matrix (us)
 GPU 0 1 2 3 4 5 6 7
 0 1.54 11.67 11.75 20.27 14.49 15.77 16.95 15.91
 1 11.67 1.48 11.44 11.59 15.22 12.17 13.20 13.18
 2 11.58 20.27 1.43 11.50 14.60 13.73 13.21 11.38
 3 11.58 20.06 12.12 1.47 13.16 16.17 14.67 11.67
 4 12.88 12.78 13.92 12.58 1.50 12.46 12.59 12.50
 5 11.48 11.93 11.78 11.88 19.18 1.42 11.59 11.40
 6 11.95 11.70 14.90 14.00 12.42 11.41 1.46 11.63
 7 12.96 11.45 11.60 11.53 20.55 11.32 12.94 1.40

 CPU 0 1 2 3 4 5 6 7
 0 3.21 9.61 9.86 9.79 10.69 10.48 10.55 10.44
 1 9.67 3.12 9.81 9.81 10.64 10.46 10.66 10.43
 2 9.89 9.62 3.20 9.98 10.96 10.67 10.75 10.72
 3 9.80 9.65 9.89 3.20 10.86 10.78 10.78 10.61
 4 10.41 10.24 10.60 10.50 3.52 11.34 11.49 11.28
 5 10.27 10.14 11.93 14.06 14.53 4.55 16.99 19.67
 6 20.68 20.61 20.39 20.63 20.86 14.35 5.21 14.46
 7 9.89 9.66 9.85 9.85 10.81 10.62 10.68 3.41

How to unlock P2P on RTX4090?

I encountered this legendary modified NVIDIA driver by tinygrad that unlocks P2P on RTX4090 GPUs which blows my mind. Let’s give it a try!

Disable IOMMU

If the following command returns an empty list, it means that the IOMMU is disabled, otherwise, you need to disable it. Consult your motherboard manual on how to disable IOMMU.

1

ll /sys/class/iommu/

Enable Resizable BAR

Enable Resizable BAR in the BIOS. It’s required for the modified NVIDIA driver to work. Consult your motherboard manual on how to enable Resizable BAR.

Uninstall the official NVIDIA driver

We will install our own modified NVIDIA driver, so we need to uninstall the official NVIDIA driver first.

1
2


nvidia-uninstall
reboot

Install the official NVIDIA driver without kernel modules

Visit the modified NVIDIA driver by tinygrad. Check out the branches with p2p in the name. There are multiple versions. You should choose one that fits. For example, I am using the 550.90.07-p2p branch. Remember the version number 550.90.07, you will need it later.

Download the official driver with the same version number as you chose (550.90.07 in my case) from the NVIDIA website. The driver I downloaded is NVIDIA-Linux-x86_64-550.90.07.run.

Install the official driver without the kernel modules because we will install our own modified kernel modules later.

1

./NVIDIA-Linux-x86_64-550.90.07.run --no-kernel-modules

Install the modified kernel modules

Clone the modified NVIDIA driver by tinygrad and install. Remember to replace the branch name with the one you chose.

1
2
3
4
5
6
7
8
9


git clone --depth=1 -b 550.90.07-p2p https://github.com/tinygrad/open-gpu-kernel-modules.git
cd open-gpu-kernel-modules
# Just in case
rmmod nvidia_drm nvidia_modeset nvidia_uvm nvidia
# Build and install the modified kernel modules
make modules -j$(nproc)
make modules_install
depmod
nvidia-smi -pm 1

Test the unlocked performance

PCI BAR

Now the BAR memory is fully available.

256MB -> 32GB.

1
2
3
4
5


# nvidia-smi -q | grep -i bar -A 3
 BAR1 Memory Usage
 Total : 32768 MiB
 Used : 24211 MiB
 Free : 8557 MiB

NCCL Tests

14.47GB/s -> 20.64GB/s: 42% improvement

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


# nThread 1 nGpus 8 minBytes 8 maxBytes 134217728 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
# Rank 0 Group 0 Pid 33895 on <REDACTED> device 0 [0x01] NVIDIA GeForce RTX 4090
# Rank 1 Group 0 Pid 33895 on <REDACTED> device 1 [0x25] NVIDIA GeForce RTX 4090
# Rank 2 Group 0 Pid 33895 on <REDACTED> device 2 [0x41] NVIDIA GeForce RTX 4090
# Rank 3 Group 0 Pid 33895 on <REDACTED> device 3 [0x61] NVIDIA GeForce RTX 4090
# Rank 4 Group 0 Pid 33895 on <REDACTED> device 4 [0x81] NVIDIA GeForce RTX 4090
# Rank 5 Group 0 Pid 33895 on <REDACTED> device 5 [0xa1] NVIDIA GeForce RTX 4090
# Rank 6 Group 0 Pid 33895 on <REDACTED> device 6 [0xc1] NVIDIA GeForce RTX 4090
# Rank 7 Group 0 Pid 33895 on <REDACTED> device 7 [0xe1] NVIDIA GeForce RTX 4090
#
# out-of-place in-place
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
 8 2 float sum -1 38.82 0.00 0.00 0 38.16 0.00 0.00 0
 16 4 float sum -1 38.31 0.00 0.00 0 38.12 0.00 0.00 0
 32 8 float sum -1 38.57 0.00 0.00 0 40.87 0.00 0.00 0
 64 16 float sum -1 40.71 0.00 0.00 0 40.65 0.00 0.00 0
 128 32 float sum -1 40.51 0.00 0.01 0 41.66 0.00 0.01 0
 256 64 float sum -1 41.63 0.01 0.01 0 41.38 0.01 0.01 0
 512 128 float sum -1 41.67 0.01 0.02 0 42.45 0.01 0.02 0
 1024 256 float sum -1 41.70 0.02 0.04 0 41.67 0.02 0.04 0
 2048 512 float sum -1 41.74 0.05 0.09 0 41.74 0.05 0.09 0
 4096 1024 float sum -1 42.35 0.10 0.17 0 42.39 0.10 0.17 0
 8192 2048 float sum -1 42.14 0.19 0.34 0 41.88 0.20 0.34 0
 16384 4096 float sum -1 43.19 0.38 0.66 0 42.73 0.38 0.67 0
 32768 8192 float sum -1 42.17 0.78 1.36 0 42.61 0.77 1.35 0
 65536 16384 float sum -1 47.58 1.38 2.41 0 46.95 1.40 2.44 0
 131072 32768 float sum -1 72.03 1.82 3.18 0 71.50 1.83 3.21 0
 262144 65536 float sum -1 119.7 2.19 3.83 0 118.6 2.21 3.87 0
 524288 131072 float sum -1 166.6 3.15 5.51 0 159.8 3.28 5.74 0
 1048576 262144 float sum -1 216.1 4.85 8.49 0 217.2 4.83 8.45 0
 2097152 524288 float sum -1 321.5 6.52 11.42 0 322.6 6.50 11.37 0
 4194304 1048576 float sum -1 478.7 8.76 15.33 0 480.1 8.74 15.29 0
 8388608 2097152 float sum -1 798.6 10.50 18.38 0 818.0 10.26 17.95 0
 16777216 4194304 float sum -1 1472.8 11.39 19.94 0 1472.1 11.40 19.94 0
 33554432 8388608 float sum -1 2870.3 11.69 20.46 0 2866.4 11.71 20.49 0
 67108864 16777216 float sum -1 5705.0 11.76 20.59 0 5695.6 11.78 20.62 0
 134217728 33554432 float sum -1 11382 11.79 20.64 0 11380 11.79 20.64 0
# Out of bounds values : 0 OK
# Avg bus bandwidth : 6.11168
#

P2P Bandwidth and Latency

P2P is now available: see the 1s in the P2P Connectivity Matrix
Bandwidth: 21.39GB/s -> 50.15GB/s: 134% improvement
Latency: 11.67us -> 1.19us: 89% reduction

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90


P2P Connectivity Matrix
 D\D 0 1 2 3 4 5 6 7
 0 1 1 1 1 1 1 1 1
 1 1 1 1 1 1 1 1 1
 2 1 1 1 1 1 1 1 1
 3 1 1 1 1 1 1 1 1
 4 1 1 1 1 1 1 1 1
 5 1 1 1 1 1 1 1 1
 6 1 1 1 1 1 1 1 1
 7 1 1 1 1 1 1 1 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
 D\D 0 1 2 3 4 5 6 7
 0 912.13 18.86 19.89 20.25 18.34 19.64 19.25 19.45
 1 20.56 920.74 19.44 20.27 18.46 19.72 19.19 19.40
 2 20.36 19.72 922.37 20.28 18.52 19.65 19.22 19.40
 3 20.43 19.68 20.04 922.49 18.47 19.59 19.15 19.41
 4 20.20 20.15 20.25 20.26 921.83 19.04 18.84 18.36
 5 20.45 20.09 20.17 20.46 18.29 922.37 18.73 18.60
 6 20.39 20.18 20.24 20.54 18.44 19.09 922.49 18.65
 7 20.53 20.14 20.12 20.51 18.17 19.11 18.74 923.46
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
 D\D 0 1 2 3 4 5 6 7
 0 913.74 25.60 25.85 25.85 21.36 22.41 21.74 22.48
 1 25.78 938.91 25.79 25.79 20.70 22.39 22.53 21.63
 2 25.70 25.95 939.00 25.78 21.42 21.59 22.21 22.16
 3 25.60 25.93 25.53 938.42 22.14 22.30 21.70 22.45
 4 22.41 21.61 22.35 22.49 940.70 25.69 25.96 25.90
 5 21.34 22.38 21.72 22.41 25.77 939.57 25.95 25.87
 6 20.57 22.27 22.32 21.60 25.75 26.01 941.73 25.83
 7 21.32 21.52 22.39 22.48 25.51 25.83 25.75 938.44
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
 D\D 0 1 2 3 4 5 6 7
 0 918.85 20.94 21.69 21.67 20.66 21.44 20.98 21.26
 1 21.48 921.29 21.57 21.54 20.20 20.65 20.87 20.70
 2 21.34 21.63 921.56 21.13 20.66 21.28 20.89 21.21
 3 21.40 21.72 21.62 921.83 20.16 20.92 20.89 20.89
 4 20.93 20.33 20.87 20.45 922.90 20.07 20.21 19.96
 5 21.40 20.86 21.26 20.90 20.23 923.69 20.18 20.33
 6 20.93 20.87 20.89 20.90 20.27 20.25 923.67 20.19
 7 21.19 20.66 21.03 20.87 20.02 20.28 20.41 923.41
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
 D\D 0 1 2 3 4 5 6 7
 0 916.45 50.10 50.46 50.29 39.90 41.85 40.83 41.68
 1 50.15 919.60 50.19 50.22 38.93 41.79 41.87 40.82
 2 49.91 50.42 920.74 50.53 40.19 40.94 41.86 41.61
 3 50.08 50.48 50.35 919.66 40.22 41.69 40.92 41.78
 4 41.60 40.94 41.86 41.93 919.93 49.80 50.36 50.22
 5 40.25 41.68 39.96 41.64 49.75 921.83 50.20 50.66
 6 39.59 41.58 41.83 40.95 49.89 50.43 921.56 50.59
 7 40.36 40.86 41.76 41.82 49.96 50.43 50.28 921.01
P2P=Disabled Latency Matrix (us)
 GPU 0 1 2 3 4 5 6 7
 0 1.50 11.60 20.17 20.18 13.08 11.27 14.58 19.33
 1 11.68 1.38 11.74 20.17 12.60 11.39 14.35 14.59
 2 11.33 11.34 1.31 11.50 16.96 20.53 16.53 20.53
 3 11.51 11.51 11.51 1.37 17.86 11.37 15.80 20.54
 4 11.47 11.79 11.44 12.57 1.38 12.24 12.93 11.73
 5 11.48 11.79 11.64 11.64 14.40 1.44 13.46 14.35
 6 13.59 15.05 13.35 12.52 12.06 11.67 1.40 12.34
 7 19.32 19.95 11.39 11.44 12.87 12.23 14.05 1.38

 CPU 0 1 2 3 4 5 6 7
 0 3.28 10.21 10.15 9.83 10.92 10.75 10.88 10.71
 1 10.07 3.14 9.81 9.56 10.58 10.41 10.61 10.35
 2 10.13 9.68 3.28 9.62 10.71 10.56 10.73 10.48
 3 9.81 9.46 9.68 3.13 10.60 10.41 10.52 10.35
 4 10.60 10.20 10.40 10.16 3.48 11.19 11.37 11.07
 5 10.37 10.02 10.29 10.04 11.10 3.49 11.24 11.00
 6 10.51 10.20 10.35 10.11 11.34 11.09 3.47 11.16
 7 10.40 10.11 10.30 10.06 11.17 11.07 11.22 3.42
P2P=Enabled Latency (P2P Writes) Matrix (us)
 GPU 0 1 2 3 4 5 6 7
 0 1.48 1.24 1.26 1.25 1.41 1.41 1.36 1.42
 1 1.19 1.38 1.18 1.18 1.29 1.42 1.29 1.26
 2 1.21 1.20 1.31 1.19 1.30 1.34 1.44 1.41
 3 1.21 1.19 1.17 1.38 1.37 1.41 1.30 1.44
 4 1.69 1.69 1.69 1.64 1.38 1.50 1.59 1.56
 5 1.64 1.60 1.60 1.62 1.50 1.43 1.56 1.48
 6 1.59 1.53 1.57 1.54 1.48 1.44 1.39 1.44
 7 1.58 1.58 1.57 1.56 1.48 1.47 1.45 1.37

 CPU 0 1 2 3 4 5 6 7
 0 3.27 2.84 2.76 2.74 2.80 2.76 2.76 2.75
 1 2.80 3.13 2.66 2.74 2.70 2.70 2.68 2.69
 2 2.83 2.80 3.26 2.85 2.85 2.78 2.77 2.76
 3 2.78 2.69 2.70 3.24 2.69 2.70 2.98 2.67
 4 3.18 3.11 3.10 3.09 3.55 3.12 3.09 3.08
 5 3.12 3.04 3.10 3.05 3.06 3.50 3.08 3.04
 6 3.20 3.11 3.18 3.11 3.11 3.16 3.57 3.08
 7 3.12 3.06 3.08 3.12 3.08 3.07 3.04 3.58

How it works? By tinygrad.

I am not an expert in such low-level hacking (but it’s soooo interesting and I want to learn). This part is from the modified NVIDIA driver by tinygrad (Thank you tinygrad). I just put it here for reference.

Normally, P2P on NVIDIA cards uses MAILBOXP2P. This is some hardware interface designed to allow GPUs to transfer memory back in the days of small BAR. It is not present or disabled in hardware on the 4090s, and that’s why P2P doesn’t work. There was a bug in early versions of the driver that reported that it did work, and it was actually sending stuff on the PCIe bus. However, because the mailbox hardware wasn’t present, these copies wouldn’t go to the right place. You could even crash the system by doing something like torch.zeros(10000,10000).cuda().to("cuda:1")

In some 3090s and all 4090s, NVIDIA added large BAR support.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


tiny@tiny14:~$ lspci -s 01:00.0 -v
01:00.0 VGA compatible controller: NVIDIA Corporation AD102 [GeForce RTX 4090] (rev a1) (prog-if 00 [VGA controller])
 Subsystem: Micro-Star International Co., Ltd. [MSI] Device 510b
 Physical Slot: 49
 Flags: bus master, fast devsel, latency 0, IRQ 377
 Memory at b2000000 (32-bit, non-prefetchable) [size=16M]
 Memory at 28800000000 (64-bit, prefetchable) [size=32G]
 Memory at 28400000000 (64-bit, prefetchable) [size=32M]
 I/O ports at 3000 [size=128]
 Expansion ROM at b3000000 [virtual] [disabled] [size=512K]
 Capabilities: <access denied>
 Kernel driver in use: nvidia
 Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

Notice how BAR1 is size 32G. In H100, they also added support for a PCIe mode that uses the BAR directly instead of the mailboxes, called BAR1P2P. So, what happens if we try to enable that on a 4090?

We do this by bypassing the HAL and calling a bunch of the GH100 methods directly. Methods like kbusEnableStaticBar1Mapping_GH100, which maps the entire VRAM into BAR1. This mostly just works, but we had to disable the use of that region in the MapAperture function for some reason. Shouldn’t matter.

1
2


[ 3491.654009] NVRM: kbusEnableStaticBar1Mapping_GH100: Static bar1 mapped offset 0x0 size 0x5e9200000
[ 3491.793389] NVRM: kbusEnableStaticBar1Mapping_GH100: Static bar1 mapped offset 0x0 size 0x5e9200000

Perfect, we now have the VRAM mapped. However, it’s not that easy to get P2P. When you run ./simpleP2P from cuda-samples, you get this error.

1
2
3
4
5
6
7
8


[ 3742.840689] NVRM: kbusCreateP2PMappingForBar1P2P_GH100: added PCIe BAR1 P2P mapping between GPU2 and GPU3
[ 3742.840762] NVRM: kbusCreateP2PMappingForBar1P2P_GH100: added PCIe BAR1 P2P mapping between GPU3 and GPU2
[ 3742.841089] NVRM: nvAssertFailed: Assertion failed: (shifted >> pField->shift) == value @ field_desc.h:272
[ 3742.841106] NVRM: nvAssertFailed: Assertion failed: (shifted & pField->maskPos) == shifted @ field_desc.h:273
[ 3742.841281] NVRM: nvAssertFailed: Assertion failed: (shifted >> pField->shift) == value @ field_desc.h:272
[ 3742.841292] NVRM: nvAssertFailed: Assertion failed: (shifted & pField->maskPos) == shifted @ field_desc.h:273
[ 3742.865948] NVRM: GPU at PCI:0000:01:00: GPU-49c7a6c9-e3a8-3b48-f0ba-171520d77dd1
[ 3742.865956] NVRM: Xid (PCI:0000:01:00): 31, pid=21804, name=simpleP2P, Ch 00000013, intr 00000000. MMU Fault: ENGINE CE3 HUBCLIENT_CE1 faulted @ 0x7f97_94000000. Fault is of type FAULT_INFO_TYPE_UNSUPPORTED_KIND ACCESS_TYPE_VIRT_WRITE

Failing with an MMU fault. So you dive into this and find that it’s using GMMU_APERTURE_PEER as the mapping type. That doesn’t seem supported in the 4090. So let’s see what types are supported, GMMU_APERTURE_VIDEO,GMMU_APERTURE_SYS_NONCOH, and GMMU_APERTURE_SYS_COH. We don’t care about being coherent with the CPU’s L2 cache, but it does have to go out the PCIe bus, so we rewrite GMMU_APERTURE_PEER to GMMU_APERTURE_SYS_NONCOH. We also no longer set the peer id that was corrupting the page table.

1
2
3
4
5
6
7


cudaMemcpyPeer / cudaMemcpy between GPU0 and GPU1: 24.21GB/s
Preparing host buffer and memcpy to GPU0...
Run kernel on GPU1, taking source data from GPU0 and writing to GPU1...
Run kernel on GPU0, taking source data from GPU1 and writing to GPU0...
Copy data back to host from GPU0 and verify results...
Verification error @ element 1: val = 0.000000, ref = 4.000000
Verification error @ element 2: val = 0.000000, ref = 8.000000

Progress! ./simpleP2P appears to work, however the copy isn’t happening. The address is likely wrong. It turns out they have a separate field for the peer address called fldAddrPeer, we change that to fldAddrSysmem. We also print out the addresses and note that the physical BAR address isn’t being added properly, they provide a field fabricBaseAddress for GMMU_APERTURE_PEER, we reuse it and put the BAR1 base address in there.

XFS Tuning for Hardware RAID

Tue, 05 Nov 2024 14:40:00 +0800

Stripe Size

When you create an hardware RAID array, you must encountered the term stripe size.

For example, an 8-disk RAID 1/0 has a stripe width of 4, with a stripe element size of 64 KB has a stripe size of 256 KB (4 * 64 KB). A 5-disk RAID 5 (4+1) with a 64 KB stripe element size also has a stripe width of 256 KB drive (4 * 64 KB) .

A stripe is the smallest chunk of data within a RAID array that can be addressed. People often also refer to this as granularity or block size. It can be compared to the blocks (logical block addressing - LBA) on conventional hard drives. Most RAID controllers allow the user to define her or his favorite stripe size, because it alters the performance characteristics of a RAID array. Reference

It is important to let the filesystem you use know the underlying RAID array’s stripe size. This is because the filesystem will align its data blocks to the RAID array’s stripe size. This will help to avoid read-modify-write operations, which can be very slow.

For example, when creating a Virtual Disk in a Dell PERC H730P Controller, you can see the stripe element size.

Tune XFS for Hardware RAID

When you format the XFS partition, you can tell XFS about the underlying array info.

For example, I have 4x8T HDDs in RAID 10 with stripe element size of 64 KB. So I have a stripe width of 2 and a stripe size of 128 KB. The corresponding XFS options is sw=2 and su=64k.

1

mkfs.xfs -f -d su=64k,sw=2 /dev/sdb

Other Resources

Share NVIDIA GPU between CTs in Proxmox VE

Mon, 04 Nov 2024 12:57:00 +0800

Background

Consider a small lab, students need to use GPU for their projects. We have a NVIDIA GPU in our Proxmox VE server, and we want to share the GPU between multiple containers so that multiple students can use the GPU at the same time.

Why not use a VM? Because a GPU can only be passed through to one VM at a time (only one student can use the GPU at a time). And resources are not flexible in VMs.

Why not create multiple users in the host and let them run their programs in the host? Because we want to isolate the students from the host, so that they can’t access the host and other students’ data.

Why not use Docker? Because Docker containers doesn’t have a full init system, and it’s hard to run some applications.

Since we use PVE and it has LXC containers built-in (called CT), it is a perfect choice.

Install Drivers on the Host

Make sure the GPU is detected by the host. Note the NVIDIA GPUs 3b:00.0 (Your address may differ).

1
2
3
4
5


# lspci | grep -i nvidia
3b:00.0 VGA compatible controller: NVIDIA Corporation TU104GL [Quadro RTX 5000] (rev a1)
3b:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)
3b:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1)
3b:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)

You may ask: does your entire lab only own one RTX 5000? What kind of lab is this? Are you cave people?

Yes, although we have multiple projects worth over millions of Chinese Yuan, most of the money is gone to the some other places (which I cannot publicly speak on the Internet 🤫 ). And the professors have no emphasis on students’ growth. As a result, we are actually poor as hell.

Since almost no one knows how to properly configure a Linux server, I want to help my classmates to learn more and let them use the only GPU. But to be honest, I won’t benefit from doing this. It’s just voluntary work.

Install prerequisites. Note that I am using pve-headers-$(uname -r) to install the headers for the current kernel. If you are using a different kernel, you may need to install the headers for that kernel. Also, you may want to use linux-headers-$(uname -r) instead of pve-headers-$(uname -r) if you are not using Proxmox VE.

1

# apt install -y gcc make pve-headers-$(uname -r)

Download CUDA toolkit from here and install it. Drivers are included in the CUDA toolkit so you don’t need to install drivers separately.

1
2


# wget <cuda-runfile-download-url>
# ./cuda_12.2.2_535.104.05_linux.run --silent

The default installation options will work fine. If anything fails, you can check the log file at /var/log/cuda-installer.log for CUDA logs and /var/log/nvidia-installer.log for NVIDIA driver logs.

PS: You need to blacklist nouveau driver. This is automatically done by PVE. If not, you can do this by creating a file /etc/modprobe.d/blacklist-nouveau.conf with the following content: blacklist nouveau. Then run update-initramfs -u to update the initramfs.

PPS: If you used to passthrough this GPU to a VM, be sure to remove the GPU from the VM’s hardware configuration in PVE otherwise PVE will bound the GPU to vfio-pci (see Kernel driver in use row in lspci -k) and cannot be used by the host.

PPPS: Some kernel versions are known to have problems with NVIDIA drivers. If you encounter problems, you may need to downgrade/upgrade the kernel. For example, kernel version 5.10.0 is known to have make[3]: *** No rule to make target 'scripts/module.lds', needed by '/tmp/selfgz38416/NVIDIA-Linux-x86_64-560.35.03/kernel-open/nvidia.ko' error.

After installation finished, check if the driver is loaded.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# nvidia-smi
Tue Nov 5 09:56:44 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Quadro RTX 5000 Off | 00000000:3B:00.0 Off | Off |
| 33% 44C P0 28W / 230W | 0MiB / 16384MiB | 6% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+

Allow NVIDIA Device Passthrough in CT

Now we need to allow the CT to access the GPU. I am using an unprivileged container here. Edit the CT’s configuration file (/etc/pve/local/lxc/<id>.conf). Add the following lines to the end of the file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


 arch: amd64
 cores: 4
 features: nesting=1
 hostname: ct-gpu-tmpl-deb127-cu122
 memory: 4096
 net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=AA:AB:F0:07:42:D0,ip=dhcp,type=veth
 ostype: debian
 rootfs: local-zfs:basevol-8001-disk-0,size=16G
 swap: 0
 unprivileged: 1
# These lines allow the container to access specific character devices (c) with rwm
# permissions (read, write, modify). These are needed for NVIDIA GPU access.
+ lxc.cgroup.devices.allow: c 195:* rwm
+ lxc.cgroup.devices.allow: c 509:* rwm
+ lxc.cgroup.devices.allow: c 235:* rwm
# These lines mount various GPU-related devices from the host into the container.
+ lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
+ lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
+ lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
+ lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
+ lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
+ lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
+ lxc.mount.entry: /dev/fb0 dev/fb0 none bind,optional,create=file

PS: If you cannot use nvidia-smi (it says Failed to initialize NVML: Unknown Error), there is a possibility that you are using cgroup2. Change all the lxc.cgroup.devices.allow lines to lxc.cgroup2.devices.allow.

Explanation:

Allows container access to NVIDIA device nodes:

c 195:* - NVIDIA character devices
c 509:* - NVIDIA UVM devices
c 235:* - NVIDIA CTL devices

Maps the following host GPU devices into container:

/dev/nvidia0 - Main GPU device
/dev/nvidiactl - NVIDIA control device
/dev/nvidia-modeset - Display mode setting
/dev/nvidia-uvm - Unified memory management
/dev/nvidia-uvm-tools - UVM diagnostic tools
/dev/dri - Direct Rendering Infrastructure
/dev/fb0 - Framebuffer device

Mount options:

bind: Mount as a bind mount
optional: Don’t fail if device doesn’t exist
create=file/dir: Create the mount point if it doesn’t exist

Note that if you are using a different GPU, you may need to change the device numbers. For example, /dev/nvidia1 instead of /dev/nvidia0. You can find the device numbers in nvidia-smi output.

Install Drivers in CT

Log into the CT. All the following commands are run in the CT.

You should be able to see NVIDIA devices inside the CT:

1
2
3
4
5
6


# ls -l /dev/nvidia*
---------- 1 root root 0 Nov 5 02:31 /dev/nvidia-modeset
crw-rw-rw- 1 nobody nogroup 507, 0 Nov 5 01:56 /dev/nvidia-uvm
crw-rw-rw- 1 nobody nogroup 507, 1 Nov 5 01:56 /dev/nvidia-uvm-tools
crw-rw-rw- 1 nobody nogroup 195, 0 Nov 5 01:56 /dev/nvidia0
crw-rw-rw- 1 nobody nogroup 195, 255 Nov 5 01:56 /dev/nvidiactl

Install CUDA and drivers, just like you would on a physical machine, except that you don’t need to install the kernel modules. I will install CUDA 12.2 (drivers are included in the CUDA installer).

1
2
3


# wget <cuda-runfile-download-url>
# apt install -y gcc
# ./cuda_12.2.2_535.104.05_linux.run --extract=$(pwd)/cu122

Note that I extracted the installer to manually install it because we want to skip kernel module installation and such options are not exposed in the installer.

Install the bundled drivers:

1
2


cd cu122
./NVIDIA-Linux-x86_64-535.104.05.run --no-nouveau-check --no-kernel-modules --silent

Run nvidia-smi to check if the driver is loaded.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# nvidia-smi
Tue Nov 5 05:49:02 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Quadro RTX 5000 Off | 00000000:3B:00.0 Off | Off |
| 33% 38C P0 23W / 230W | 0MiB / 16384MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+

We can now see that the GPU is accessible in the CT.

Let’s continue with the CUDA installation. Remember to uncheck the driver installation option because we have already installed the drivers above.

1

./cuda-linux.12.2.2-535.104.05.run

After a successful installation, you should add cuda binaries to PATH. Instructions should be printed at the end of the installation. Then you can run nvcc to see if CUDA is installed correctly.

Everything should be working by this point.

Missing `nvidia-uvm` and High Idle Power Draw

One problem I encountered is that when the host reboots, the GPU is not accessible in the CT. This is because nvidia-uvm device isn’t created until an application attempts to interact with the graphics card. This is a problem because no application will interact with the GPU at boot, so no nvidia-uvm device is created. But the CT needs the nvidia-uvm device bind-mounted at CT-startup in order to access the GPU.

Also, the graphics card have insanely high power draw at idle (over 100 Watts). The GPU is in P0 and never leaves it. We can use nvidia-persistenced to let the GPU enter a low-power state (P8) when not in use.

To solve this, we can run nvidia-smi -pm 1 (which enables nvidia-persistenced and keeps nvidia character device and handles frequency scaling) at boot. Add the following line to the host’s crontab to run nvidia-smi -pm 1 at boot.

PS: This only works if the host is a headless server (no monitor attached). If you have a monitor attached, you may need to run nvidia-smi below instead.

1
2


# crontab -e
@reboot /usr/bin/nvidia-smi -pm 1

Downsides

Despite the fact that this method works best for us, there are some downsides:

The CT will have full access to the GPU. If one CT uses all the GPU memory, other CTs will be starving. So you must trust the users of the CTs. This is not a problem for us because we know each other.
Driver updates are a bit more complicated. You need to update the drivers on the host and in all of the CTs. It’s best to not update the drivers too often.
The CTs share the same kernel with the host. To avoid potential compatibility issues, we don’t update the kernel unless necessary.

Fake macOS Version Detected by Homebrew

Tue, 08 Oct 2024 22:08:00 +0800

Background

One particular issue that I encountered when I tried to install some software using Homebrew on my Mac is that Homebrew deprecates support for older macOS versions just like Apple do. Once a macOS version is deprecated, Homebrew will not provide bottles (precompiled binaries) for that version. This means that you have to compile the software from source, which is extremely time-consuming and sometimes error-prone. Imagine installing shell-check will require to build ghc from source, which takes hours, and then use ghc to build shell-check from source. This is not a good experience.

Solution

Pin homebrew/core to an older version

One way to work around this issue is to use a older version of the homebrew/core tap, such that the bottles are still available for the macOS version you are using because they are built when your macOS version is supported. The downside is that you will be using outdated software. If this is not an issue for you then congratulations. You can achieve this by running the following command:

1
2
3


$ brew tap homebrew/core
$ cd $(brew --repo homebrew/core)
$ git reset --hard <commit-hash>

Make sure you disable the auto-update of Homebrew so that it does not update the homebrew/core tap to the latest version. You can do this by running the following command:

1

$ echo 'export HOMEBREW_NO_AUTO_UPDATE=1' >> ~/.zshrc

Fake macOS version

Another way is to fake the macOS version that Homebrew detects. This way, Homebrew will provide bottles for the macOS version you are faking. Most software will work and you will be using the latest software. However, this may not work for all software as some software may have dependencies that are not available for the macOS version you are faking. You can achieve this by running the following command:

1
2


$ export HOMEBREW_FAKE_MACOS=13.0 # Choose a macOS version that is supported by Homebrew and close to your macOS version
$ brew install xxx

Explanation

Homebrew actually has an private undocumented API that allows you to fake the macOS version. https://github.com/Homebrew/brew/blob/a3d8f4e0e4a22da9990d59cca70bec1e7be726cf/Library/Homebrew/os/mac.rb#L41

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# This can be compared to numerics, strings, or symbols
# using the standard Ruby Comparable methods.
#
# @api internal
sig { returns(MacOSVersion) }
def self.full_version
 @full_version ||= if (fake_macos = ENV.fetch("HOMEBREW_FAKE_MACOS", nil)) # for Portable Ruby building
 MacOSVersion.new(fake_macos)
 else
 MacOSVersion.new(VERSION)
 end
end

Example

I have macOS Monterey (12.0) running, which is just deprecated by Homebrew. I want to install batt which is not available as a bottle for macOS Monterey.

If I run brew install batt, I will get the following error:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


$ brew install batt
Warning: You are using macOS 12.
We (and Apple) do not provide support for this old version.
It is expected behaviour that some formulae will fail to build in this old version.
It is expected behaviour that Homebrew will be buggy and slow.
Do not create any issues about this on Homebrew's GitHub repositories.
Do not create any issues even if you think this message is unrelated.
Any opened issues will be immediately closed without response.
Do not ask for help from Homebrew or its maintainers on social media.
You may ask for help in Homebrew's discussions but are unlikely to receive a response.
Try to figure out the problem yourself and submit a fix as a pull request.
We will review it but may or may not accept it.

It’s Oct 8, 2024 and the latest macOS version is macOS Sequoia (15.0). So the last 3 supported macOS versions are:

macOS Sequoia (15.0)
macOS Sonoma (14.0)
macOS Ventura (13.0)

You can confirm that by checking out batt’s formula:

1
2
3
4
5
6


$ brew edit batt
...
sha256 cellar: :any_skip_relocation, arm64_sequoia: "61bd7790a82f2269b9a0ce1585c57564d03be4e4ac89b8f0d8843c0073a688e6"
sha256 cellar: :any_skip_relocation, arm64_sonoma: "198bd7bb9a808f0a9e4cb1a31b7e9c2a72d690ba1d07ebddf78b6d0ce0b6dd03"
sha256 cellar: :any_skip_relocation, arm64_ventura: "6eff598159b263327b8b562ab32f5e5e7157c20f25cbecfa08b48eda794c4c43"
...

Showing only the bottles for macOS Sequoia (15.0), macOS Sonoma (14.0), and macOS Ventura (13.0) are available.

Sadly my macOS version (Monterey 12.0) is not in the list, but macOS Ventura (13.0) is the closest to macOS Monterey (12.0). So I can fake the macOS version to macOS Ventura (13.0) by running the following command:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


$ export HOMEBREW_FAKE_MACOS=13.0
$ brew config
...
macOS: 13.0-arm64 # YES! I'm now faking macOS Ventura (13.0)
...
$ brew install batt
==> Downloading https://ghcr.io/v2/homebrew/core/batt/manifests/0.3.1
Already downloaded: /Users/charlie/Library/Caches/Homebrew/downloads/86da7d77f0bacb44475e0280eef162148e3b51df8ea44ff9bc16a5a7bba6d39f--batt-0.3.1.bottle_manifest.json
==> Fetching batt
==> Downloading https://ghcr.io/v2/homebrew/core/batt/blobs/sha256:6eff598159b263327b8b562ab32f5e5e7157c20f25cbecfa08b48eda794c4c43
Already downloaded: /Users/charlie/Library/Caches/Homebrew/downloads/24fc002de2ee11096a544dd84ff1033262a5cb627c2b824c915a552bb6a588dd--batt--0.3.1.arm64_ventura.bottle.tar.gz

As you can see, Homebrew is downloading the bottle for macOS Ventura (13.0) successfully! (Of course, the log is showing that I have already downloaded earlier, but you get the idea.)

batt itself does not rely on any APIs that are not available in my current macOS version, so it works perfectly fine.

What happens when your Kubernetes master has low memory or slow disk?

Thu, 15 Aug 2024 14:54:00 +0800

Background

We want to test some sandbox application on our new cluster. My colleague reported that the performance is not as expected and wanted me to take a look.

The sandbox application works like this: it starts a python container, runs some scripts, and exits. They executes scripts provided by the user, so the cluster is expected to create and destroy a lot of containers in a short period of time.

Assumptions

Considering the characteristics of the application, the bottleneck could be the following:

CPU/Memory of the worker nodes, or the container runtime
Performance of the master node (apiserver, etcd, and etc)

Since we have relatively beefy worker nodes, each with 2*Xeon Platinum 8353v (72C144T) and 512GB of memory, the bottleneck is likely the master node (8C 16GB memory).

Let’s verify this assumption.

Observations

I ran k6 to load test the cluster.

Pods started to pending…

Hmm, it smelled like a etcd issue (the newly-created Pods is not able to be written to the database). I looked at the etcd dashboard. Sure enough, db backend is having some trouble writing the files.

I grabed the the disk stats: the disks on the master node is almost fully utilized! A better disk (SSD) is really needed.

Side note

We also experienced unexpected issue when I applied a cilium config, the whole cluster goes down (Thanks god. This is not a production cluster, or this will be a total disaster). At first, I assumed the cilium config that I written is wrong and I reverted, hoping the cluster will be fixed. Nope, all nodes are down. However, some interesting behavior is noticed: all worker nodes are in a ready - notready loop.

Upon further investigation, I noticed the apiserver is constantly being OOM killed.

The apiserver is using more memory than the node has.

It seems something is hitting the apiserver really hard. Under normal circumstances, I will go to the master node to look at the apiserver logs. However, the master node is down (because OOM) and I cannot ssh into it.

The cluster is stuck in such a loop:

Worker nodes are all starting (cilium starting) and pulling/changed config through apiserver
apiserver on the master node OOM’ed because of bursts of requests
the apiserver is OOM killed
worker node cannot report status to the apiserver
worker node is marked as not ready; cilium pods are terminated because of nodes are not ready
go to step 1

Since the cluster cannot properly start in such a scenario, I have to use some dirty fix to start the cluster: just after the apiserver is started, and before the apiserver eats all memory, such that the master node is still alive, quickly SSH into the master node and kill the apiserver process. By doing this a few times, we can span the burst of requests over a longer period of time, avoiding overloading the apiserver (causing it to be killed).

Solution

The solution is simple: replace the master nodes with faster ones (much more memory, NVME SSDs).

Rename RDMA NIC Interface Names

Fri, 09 Aug 2024 14:28:00 +0800

Background

The RDMA NIC names (InfiniBand HCAs in this case) of our nodes are inconsistent. We want to rename them to a consistent naming scheme.

One node:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81


root@tj01-h20-node139:/# ibstatus
Infiniband device 'ibp171s0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x70
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'ibp187s0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x6f
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'ibp203s0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x29
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'ibp219s0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x6a
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'ibp41s0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x6c
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'ibp59s0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x2b
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'ibp75s0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x80
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'ibp93s0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x6e
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'rocep22s0f0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x0
 sm lid: 0x0
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 25 Gb/sec (1X EDR)
 link_layer: Ethernet

Another node:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81


root@tj01-h20-node140:/# ibstatus
Infiniband device 'mlx5_2' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x68
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'mlx5_3' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x71
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'mlx5_4' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x27
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'mlx5_5' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x69
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'mlx5_6' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x6d
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'mlx5_7' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x6b
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'mlx5_8' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x2a
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'mlx5_9' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x2c
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'mlx5_bond_0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x0
 sm lid: 0x0
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 25 Gb/sec (1X EDR)
 link_layer: Ethernet

Note that one node has the names ibp* and the other has mlx5_*. We want to make node 1 follow a same naming scheme as the other nodes.

To be fair ibp* is actually consistent (named using PCI location). mlx5_* is not consistent, which depends on the card initialization order and can change after a reboot.

Anyway, since all other nodes are using mlx5_* naming scheme, we want to rename all of them to mlx5_* for consistency with the other nodes.

Solution

1

cp cp /lib/udev/rules.d/60-rdma-persistent-naming.rules /etc/udev/rules.d/

This is all you need. Now reboot your server.

If you are interested in the details, read on.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
# Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
#
# Rename modes:
# NAME_FALLBACK - Try to name devices in the following order:
# by-pci -> by-guid -> kernel
# NAME_KERNEL - leave name as kernel provided
# NAME_PCI - based on PCI/slot/function location
# NAME_GUID - based on system image GUID
# NAME_FIXED - rename the device to the fixed named in the next argument
#
# The stable names are combination of device type technology and rename mode.
# Infiniband - ib*
# RoCE - roce*
# iWARP - iw*
# OPA - opa*
# Default (unknown protocol) - rdma*
#
# Example:
# * NAME_PCI
# pci = 0000:00:0c.4
# Device type = IB
# mlx5_0 -> ibp0s12f4
# * NAME_GUID
# GUID = 5254:00c0:fe12:3455
# Device type = RoCE
# mlx5_0 -> rocex525400c0fe123455
#
ACTION=="add", SUBSYSTEM=="infiniband", PROGRAM="rdma_rename %k NAME_KERNEL"

# Example:
# * NAME_FIXED
# fixed name for specific board_id
#
#ACTION=="add", ATTR{board_id}=="MSF0010110035", SUBSYSTEM=="infiniband", PROGRAM="rdma_rename %k NAME_FIXED myib"

Look at the line contains PROGRAM="rdma_rename %k NAME_KERNEL". So if you want to use mlx5_* (called kernel names), you can use NAME_KERNEL as the rename mode (default). If you want to use ibp* (called PCI names), you can use NAME_PCI as the rename mode or use NAME_FALLBACK, which first tries NAME_PCI.

NFS over RDMA Setup & Benchmark

Fri, 19 Jul 2024 22:36:00 +0800

So I decided to test NFS over RDMA somehow.

Hostname	`ft2000` (Server)	`t3640` (Client)
CPU	Phytium FT-2000+/64	Intel Core i9-10900K
RAM	Quad Channel DDR4 128GB	Dual Channel DDR4 64GB
NIC	Mellanox ConnextX-4 Lx 25GbE	Mellanox ConnextX-4 Lx 25GbE
OS	UOS Server 20 1070a	Debian 12

Two servers are connected using a 25GbE network. The goal is to set up NFS over RDMA and benchmark it to see the performance difference between NFS over TCP and NFS over RDMA.

RDMA Link Verification

Install RDMA/IB dependencies:

Dependencies should be the same on both devices (no server and client differences). The difference is only the package manager (and OS).

If you are using Mellanox OFED, you can skip this step. Be sure to install Mellanox OFED using --with-nfsrdma flag. Otherwise, you will not be able to use NFS over RDMA.

On UOS Server 20 1070a, which is Anolis8-based, which is again CentOS8-based:

1
2
3
4


root@ft2000:/# yum install -y libibverbs librdmacm opensm-libs rdma-core rdma-core-devel \
 librdmacm-utils opensm-static srp_daemon ucx-devel ucx-rdmacm infiniband-diags ibacm \
 opensm-devel ucx ucx-ib libibumad opensm mstflint ucx-cma openmpi rshim libibverbs-utils \
 perftest rdma-core rdma-core-devel infiniband-diags rshim

On Debian 12:

1
2
3
4


root@t3640:/# apt install -y infiniband-diags srptools perftest opensm-doc librdmacm-dev \
 rdmacm-utils librdmacm1 ibacm libibmad-dev libibmad5 libibumad-dev libibumad3 \
 ibverbs-utils libibverbs-dev libibverbs1 mstflint rdma-core opensm fio librbd1 \
 librados2 libibnetdisc5 ibverbs-providers

Make sure link is up on both servers using ip or ibstatus command:

Client:

Use ip (note state UP):

1
2


root@t3640:/# ip l show dev enp1s0f1np1
6: enp1s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

Or use ibstatis (note phys state: 5: LinkUp):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


root@t3640:/# ibstatus
Infiniband device 'mlx5_0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x0
 sm lid: 0x0
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 10 Gb/sec (1X QDR)
 link_layer: Ethernet

Infiniband device 'mlx5_1' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x0
 sm lid: 0x0
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 25 Gb/sec (1X EDR)
 link_layer: Ethernet

Server:

Use ip (note state UP):

1
2


root@ft2000:/# ip l show dev enp16s0f0np0
4: enp16s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

Or use ibstatis (note phys state: 5: LinkUp):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


root@ft2000:/# ibstatus
Infiniband device 'mlx5_0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x0
 sm lid: 0x0
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 25 Gb/sec (1X EDR)
 link_layer: Ethernet

Infiniband device 'mlx5_1' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x0
 sm lid: 0x0
 state: 1: DOWN
 phys state: 3: Disabled
 rate: 40 Gb/sec (4X QDR)
 link_layer: Ethernet

Verify IP connectivity between the servers (ping the IP address of the other server):

1
2
3
4


root@t3640:/# ping 192.168.23.1
PING 192.168.23.1 (192.168.23.1) 56(84) bytes of data.
64 bytes from 192.168.23.1: icmp_seq=1 ttl=64 time=0.199 ms
64 bytes from 192.168.23.1: icmp_seq=2 ttl=64 time=0.206 ms

Make sure the the InfiniBand kernel modules are enabled.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# On both servers
root@ft2000:/# lsmod | grep ^ib
ib_srpt 262144 0
ib_isert 262144 0
ib_iser 262144 0
ib_umad 262144 0
ib_ipoib 327680 0
ib_cm 262144 3 rdma_cm,ib_ipoib,ib_srpt
ib_uverbs 327680 2 rdma_ucm,mlx5_ib
ib_core 458752 12 rdma_cm,ib_ipoib,rpcrdma,ib_srpt,iw_cm,ib_iser,ib_umad,ib_isert,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm

Make sure you have a lossless network:

In case the RDMA is running over Ethernet (RoCE) you need to make sure that the network is configured to be loss-less, which means that either flow control (FC) or priority flow control PFC is enabled on the adapter ports and the switch.

In case of lab environment or small setup, you can use Global Pause Flow Control to create loss-less environment. To check what is the global pause configuration use the following command (by default it is enabled normally).

1
2
3
4
5


root@t3640:/# ethtool -a enp1s0f0np0
Pause parameters for enp1s0f0np0:
Autonegotiate: off
RX: on
TX: on

In case it is disabled, run:

1

root@t3640:/# ethtool -A enp1s0f0np0 rx on tx on

Test RDMA speed. Refer to my previous blog InfiniBand Performance Test for details.

Run the following command on client and server, respectively:

1
2


root@t3640:/# ib_send_bw --report_gbit -F -d mlx5_1
# device name can be seen in `ibstatus` command

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


root@ft2000:/# ib_send_bw --report_gbit -a -F -d mlx5_0 192.168.23.2
---------------------------------------------------------------------------------------
 Send BW Test
 Dual-port : OFF Device : mlx5_0
 Number of qps : 1 Transport type : IB
 Connection type : RC Using SRQ : OFF
 PCIe relax order: ON
 ibv_wr* API : ON
 TX depth : 128
 CQ Moderation : 100
 Mtu : 1024[B]
 Link type : Ethernet
 GID index : 3
 Max inline data : 0[B]
 rdma_cm QPs : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0087 PSN 0x176d37
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:23:01
 remote address: LID 0000 QPN 0x0188 PSN 0x789e84
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:23:02
---------------------------------------------------------------------------------------
 #bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
 65536 1000 23.14 23.13 0.044119
---------------------------------------------------------------------------------------

As you can see, the speed is around 23 Gb/sec. This is the maximum speed of the network (25G). If you see a lower speed, you need to check the network configuration and the switch configuration.

Configure NFS without RDMA

Just to make sure NFS is working.

Install NFS on server:

1
2
3


root@ft2000:/# yum -y install nfs-utils # On Debian: apt install nfs-kernel-server
root@ft2000:/# systemctl start nfs-server rpcbind
root@ft2000:/# systemctl enable nfs-server rpcbind

Use /tmp as a test dir and share it:

Since I want to test the network performance (TCP and RDMA), I don’t want to be bottlenecked by the disk speed. So I will use /tmp, which is backed by RAM (usually), as a reasonably fast directory to avoid bottlenecks.

Make sure it is tmpfs, otherwise IO will be limited by your disk speed.
1
2
root@ft2000:/# mount | grep /tmp
tmpfs on /tmp type tmpfs (rw,nosuid,nodev)

1

root@ft2000:/# echo '/tmp *(rw,async,insecure,no_root_squash)' >>/etc/exports

Make it available:

1
2


root@ft2000:/# exportfs -a
root@ft2000:/# systemctl restart nfs-server

Install NFS on client:

1

root@t3640:/# apt install nfs-common

Mount the NFS share temporarily:

1
2


root@t3640:/# mkdir -p /mnt/nfs
root@t3640:/# mount -t nfs 192.168.23.1:/tmp /mnt/nfs

Verify the mount:

Should see the files in /tmp on the server.

1

root@t3640:/# ls -l /mnt/nfs

Configure NFS with RDMA

Load RDMA transport module on the server:

1
2
3


root@ft2000:/# modprobe svcrdma
# To make it permanent: echo 'svcrdma' >>/etc/modules
root@ft2000:/# systemctl restart nfs-server

Instruct the server to listen on the RDMA transport port (20049 is the default port):

Note: if you see echo: write error: protocol not supported, it means that the NFSoRDMA is not supported in the Mellanox OFED. You need to use the inbox OS driver (refer to Install RDMA/IB dependencies section in this blog) or re-install Mellanox OFED with --with-nfsrdma flag. I was tripped by this issue.

1
2


root@ft2000:/# echo rdma 20049 > /proc/fs/nfsd/portlist
# To make it permanent: echo 'rdma=nfsrdma' >> /etc/nfs.conf

Load the RDMA transport module on the client

1
2
3


root@t3640:/# modprobe xprtrdma
# To make it permanent: echo 'xprtrdma' >>/etc/modules
root@t3640:/# systemctl restart nfs-utils.service

Mount the NFS share with RDMA:

Before running the mount command, unmount the NFS share if it is mounted (umount /mnt/nfs).

1
2


root@t3640:/# mount -t nfs -o rdma,port=20049 192.168.23.1:/tmp /mnt/nfs
# To make it permanent: echo '192.168.23.1:/tmp /mnt/nfs nfs defaults,proto=rdma,port=20049 0 0' >> /etc/fstab

Check the mount parameters:

If you see proto=rdma, it means the NFS is mounted using RDMA.

1
2
3


root@t3640:/# nfsstat -m
/mnt/nfs from 192.168.23.1:/tmp
 Flags: rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.23.2,local_lock=none,addr=192.168.23.1

Verify the mount:

Should see the files in /tmp on the server.

1

root@t3640:/# ls -l /mnt/nfs

Benchmark

Use fio to benchmark the NFS performance. Parameters such as bs, iodepth will be changed to different values later to evaluate the performance in different scenarios.

I will only test random read performance since sequential read performance is not a main concern even using TCP let alone RDMA.

The actual command will be:

1
2
3


root@t3640:/# fio --rw=randread --bs=4k --numjobs=1 --iodepth=1 --runtime=10 --time_based --loops=1 \
 --ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap \
 --exitall --name task1 --filename=/mnt/nfs/testfile --size=256M

The full log and related scripts can be found in the Appendix.

Here are the results we have all been waiting for!

For IO operations per second (IOPS), the average IOPS for NFS over RDMA is higher than NFS over TCP. The difference is more significant when the block size is larger and the most significant when iodepth=12. We can also see that no matter the block size and iodepth, the IOPS for NFS over TCP is capped at around 47 KIOPS, while the one of RDMA is significantly higher at over 180 KIOPS.

The reason why the difference is smaller with larger block sizes is that the network bandwidth is the bottleneck. For our 25GbE network, the maximum bandwidth is around 2800 MiB/s. We are hitting this limit with larger block sizes and this much IOPS. This is unrelated to the RDMA or TCP. If you have a faster network, you will see this difference for larger block sizes as well.

For 4K block size:

IO Depth	KIOPS (RDMA)	KIOPS (TCP)	Ratio
8	124.5	36.1	3.4
12	156.0	38.1	4.1
16	165.8	41.4	4.0
24	177.0	45.8	3.9
32	179.4	50.7	3.5

For bandwidth, the average bandwidth for NFS over RDMA is significantly higher (around 4x for small block sizes).

The reason why the difference is more significant when the block size is small (4-16) is that we are limited by the network bandwidth. Since we have a 25GbE network, the maximum bandwidth is around 2800 MiB/s so you can see that the bandwidth is capped at 2800 MiB/s for large block sizes and deep IO depths. If you have a faster network, you will see this 4x difference for larger block sizes as well.

Latency-wise, the average latency for NFS over RDMA is slightly lower than NFS over TCP. The difference is more significant for small block size and large iodepths.

Appendix

Benchmark script benchmark.sh: automatically run fio with different bs and iodepth values and saves logs (it saves stdout, not the fio-generated log, e.g. --bandwidth-log).
Plotting script plot.py: parses the fio logs (stdout) saved by benchmark.sh and plots the results, which is the three figures you see above in the Benchmark section.

Raw fio Logs (Click to Show)

Flash 10G CX4121A to 25G

Fri, 19 Jul 2024 16:37:00 +0800

10G ConnectX-4 Lx NICs can be flashed to 25G with the right firmware. This is useful if you have a 10G NIC and want to upgrade to 25G without buying a new NIC. This article will guide you through the process of flashing a 10G Mellanox ConnectX-4 Lx NIC to 25G.

Warning: Do not try this on MCX4121C. It will brick the NIC.

Download firmware from Firmware for ConnectX®-4 Lx EN. Choose MCX4121A-ACUT. I will assume the firmware file is named fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACU_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin.

If you are wondering what’s the difference between models, here’s a table for ConnectX-4 Lx cards (original document). You should choose 25G models to flash to.

Max Network Speed	Interface Type	Supported Ethernet Speeds (GbE)	Host Interface	Additional Features	OPN
1x 10GbE	SFP28	10, 1	PCIe 3.0 x8		MCX4111A-XCAT
2x 10GbE	SFP28	10, 1	PCIe 3.0 x8		MCX4121A-XCAT
2x 10GbE	SFP28	10, 1	PCIe 3.0 x8	Host Management, UEFI Enabled	MCX4121A-XCHT
1x 25GbE	SFP28	25, 10, 1	PCIe 3.0 x8		MCX4111A-ACAT
1x 25GbE	SFP28	25, 10, 1	PCIe 3.0 x8	UEFI Enabled	MCX4111A-ACUT
2x 25GbE	SFP28	25, 10, 1	PCIe 3.0 x8		MCX4121A-ACAT
2x 25GbE	SFP28	25, 10, 1	PCIe 3.0 x8	UEFI Enabled	MCX4121A-ACUT
1x 40GbE	QSFP28	40, 25, 10, 1	PCIe 3.0 x8		MCX4131A-BCAT
1x 50GbE	QSFP28	50, 40, 25, 10, 1	PCIe 3.0 x8		MCX4131A-GCAT

Download and install NVIDIA Firmware Tools (MFT).
mst start
mst ststus You will see you MST device, e.g. /dev/mst/mt4117_pciconf0. I will use this device path in this article. If your path differs from this, please change it accordingly.
Save GUID, MAC: flint -d /dev/mst/mt4117_pciconf0 query full > flint_query.txt
Save hardware info: flint -d /dev/mst/mt4117_pciconf0 hw query > flint_hwinfo.txt
Save current firmware: flint -d /dev/mst/mt4117_pciconf0 ri orig_firmware.bin
Save current firmware config: flint -d /dev/mst/mt4117_pciconf0 dc orig_firmware.ini
Save current PXE ROM (if exists): flint -d /dev/mst/mt4117_pciconf0 rrom orig_rom.bin
Save current PCI VPD: mlxburn -d /dev/mst/mt4117_pciconf0 -vpd > orig_vpd.txt
flint -i fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACU_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin verify
flint -i fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACU_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin -d /dev/mst/mt4117_pciconf0 -allow_psid_change burn

Reboot you machine. You should see your NIC is now have 25G speed.

Bonus:

To change MAC address: flint -d /dev/mst/mt4117_pciconf0 -mac 02c90abcdef0 sg
To change GUID: flint -d /dev/mst/mt4117_pciconf0 -guid 0002c9000abcdef0 sg

InfiniBand Performance Test

Fri, 19 Jul 2024 11:10:00 +0800

We have a 200Gbps InfiniBand network among about 100 nodes, connected using NVIDIA ConnectX-7 NICs (HDR200) to a NVIDIA MQM9700 switch. Each OSFP twin port (2xNDR, 2x400Gbps) on QM9700 is splitted into 4xNDR200 ports to connect to 4xNVIDIA ConnectX-7 NICs using 2xNDR to 4xNDR200 DAC/ACC (OSFP to 4x OSFP).

We want to test the performance of the network between 2 nodes just to make sure it is working. We expect to see a bandwidth of around 200Gbps.

Check the status of the Infiniband devices. Make sure at least one link is up (phys state: 5: LinkUp) on both nodes.

1
2
3
4
5
6
7
8
9


root@tj01-h20-node139:/# ibstatus
Infiniband device 'mlx5_7' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x6c
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Check if both nodes are connected to the Subnet Manager. You should see the name of both nodes appear on the list.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


root@tj01-h20-node139:/# ibnetdiscover
#
# Topology file: generated on Fri Jul 19 02:49:25 2024
#
# Initiated from node <redacted> port <redacted>

vendid=0x2c9
devid=0xd2f2
sysimgguid=0x9c05<redacted>
switchguid=0x9c05<redacted>(9c05<redacted>)
Switch 129 "S-9c05<redacted>" # "MF0;CNTSN-POD229-QM97-IB-02:MQM9700/U1" enhanced port 0 lid 1 lmc 0
[1] "H-b83f<redacted>"[1](b83f<redacted>) # "tj01-4090-node005 mlx5_2" lid 22 2xNDR
[2] "H-946d<redacted>"[1](946d<redacted>) # "tj01-4090-node004 mlx5_2" lid 27 2xNDR
[3] "H-a088<redacted>"[1](a088<redacted>) # "tj01-4090-node006 mlx5_2" lid 10 2xNDR
[4] "H-946d<redacted>"[1](946d<redacted>) # "tj01-4090-node007 mlx5_2" lid 12 2xNDR
[6] "H-946d<redacted>"[1](946d<redacted>) # "tj01-4090-node009 mlx5_2" lid 11 2xNDR
...omitted

Make sure perftest is installed.

On one node:

1

root@tj01-4090-node099:/# ib_send_bw --report_gbit -a -F -d mlx5_2

--report_gbit: Show result in gigabit.
-a: Run sizes from 2 till 2^23.
-F: Do not show a warning even if cpufreq_ondemand module is loaded, and cpu-freq is not on max.
-d: Use IB device. To make sure you are using the device you want to test (if you have multiple IB devices like me).

On another node:

1

root@tj01-h20-node139:/# ib_send_bw --report_gbit -a -F -d mlx5_2 tj01-4090-node099

tj01-4090-node099 is the IP address (hostname) of the other node.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


---------------------------------------------------------------------------------------
 Send BW Test
 Dual-port : OFF Device : mlx5_2
 Number of qps : 1 Transport type : IB
 Connection type : RC Using SRQ : OFF
 PCIe relax order: ON
 ibv_wr* API : ON
 TX depth : 128
 CQ Moderation : 100
 Mtu : 4096[B]
 Link type : IB
 Max inline data : 0[B]
 rdma_cm QPs : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x6c QPN 0x0050 PSN 0xddd611
 remote address: LID 0x28 QPN 0x0050 PSN 0xa3c746
---------------------------------------------------------------------------------------
 #bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
 2 1000 0.052812 0.050794 3.174618
 4 1000 0.080899 0.080547 2.517093
 8 1000 0.17 0.17 2.624748
 16 1000 0.35 0.35 2.741373
 32 1000 0.70 0.70 2.736427
 64 1000 1.62 1.62 3.159528
 128 1000 3.07 3.07 2.994503
 256 1000 5.46 5.46 2.663988
 512 1000 13.24 13.23 3.230121
 1024 1000 26.58 26.56 3.241834
 2048 1000 51.80 51.75 3.158744
 4096 1000 82.15 82.11 2.505647
 8192 1000 89.37 89.33 1.363014
 16384 1000 89.01 87.24 0.665597
 32768 1000 172.67 96.54 0.368268
 65536 1000 181.95 108.83 0.207586
 131072 1000 181.82 137.78 0.131399
 262144 1000 187.32 163.15 0.077797
 524288 1000 190.12 179.08 0.042696
 1048576 1000 191.18 187.28 0.022326
 2097152 1000 192.69 192.69 0.011485
 4194304 1000 195.53 195.53 0.005827
 8388608 1000 196.69 196.69 0.002931
---------------------------------------------------------------------------------------

macOS Nix First Taste

Thu, 27 Jun 2024 15:13:00 +0800

Note

Note that this blog currently acts as a reference for me. It is not a comprehensive guide on why and how to use Nix on macOS.

I chose nix because I am tired of Homebrew’s slowness and NOT PROVIDING BOTTLES (BINARY) FOR OLD MACOS VERSIONS.

Install

I am not using the official installer because it cannot survive macOS updates. Instead, I am using the Determinate System’s Nix Installer.

1
2
3
4
5
6
7


ARCH=$(uname -m | sed 's/arm64/aarch64/')
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
curl -L -o nix-installer https://github.com/DeterminateSystems/nix-installer/releases/latest/download/nix-installer-$ARCH-$OS
chmod +x nix-installer
# Optionally, move nix-installer to your PATH
# sudo install nix-installer /usr/local/bin
./nix-installer install --explain

Pin Nixpkgs

I don’t want to download a tarball and extract it every now and then. I know you can set tarball-ttl to a higher value, but I don’t want to do that either. Just pin nixpkgs.

1

nix registry pin nixpkgs

If you are like me in China, you may want to use a mirror.

1

echo "substituters = https://mirrors.sjtug.sjtu.edu.cn/nix-channels/store/ https://cache.nixos.org/" | sudo tee -a /etc/nix/nix.conf

Using

I wrote some scripts just to make my life easier.

Later Steps

There is nix-darwin to make declarative configuration easier. I don’t have time to try it yet but I will definitely give it a try when I have time.

高速做种指南

Wed, 19 Jun 2024 22:11:00 +0800

前言

由于最近接入了 10Gbps 网络，发现在 PT 做种时，传输速度并没有达到预期的速度，于是优化了一波。本文记录了一些针对高速做种的建议、注意点和技巧。

硬件条件

足够快的网络

做种本质上就是网络传输，快速的网络是做种的基础。

确保上传带宽

做种嘛，主要是给别人传输数据，吃的是你的上传带宽，跟下载几乎没有关系，这是最重要的。例如原本 50Mbps 的上传情况下，买一条 100Mbps 的宽带比以下所有的优化都重要。

但是在中国，一般来说家庭宽带的上行速度都是比较低的。例如，电信的 1000M 宽带，上行速度一般不会超过 100Mbps，通常在 50Mbps 左右。这样的话，你的上传速度就被限制在 6.25MB/s 左右，在这个速度下，其实最主要的瓶颈在于你的上传带宽上，不太需要下面说的硬件/软件优化🫠。你可能需要找不少资源来找到高上传带宽的宽带，例如某些地区的移动宽带会给 IPv6 相当大的上传。你还要注意如何不被运营商认为是 PCDN 而封你的号。

由于本文聊的是高速做种，针对的是接近 1Gbps 的上传带宽的用户。国外的情况暂且不讨论（高速宽带价格更低，很容易购买到超过 1Gbps 的云服务器、seeding box 等），在国内，拥有高速上传带宽的用户主要有几种情况：

企业用户（例如大型公司、数据中心等），使用价格高昂，上下对等（即下载和上传速度一样快），租云服务器也属于这个范畴；
高校用户，使用价格较低（对用户来说，对学校来说不是这回事），上下对等；
其他特殊用户，例如某些地区的移动家庭宽带，500M 宽带上行可以达到 200M。

如果你是这些用户，那么你的上传带宽可能会达到 1Gbps 甚至更高，这时候就需要考虑下面的硬件/软件优化了。不过估计阅读本文的人大部分是高校教育网用户，低廉的价格超大的上传带宽（

在本文中，我的接入网络为上下 10Gbps 的教育网 CERNET 对等接入。

外界可联通

大部分情况下，你做种时是别人主动连接你的，所以你的网络需要是可被连接的。

对于 IPv4 来说：

最好的情况是你拥有一个独立的公网 IP ，这是最简单的情况。如果公网 IP 在你的网卡上，那么你不需要做任何设置。如果公网 IP 在你家的光猫上，那么通过简单的端口映射/UPnP 等就能完成。这种情况国内不常见。
其次如果你的网络是 NAT1（全锥形 NAT）的（指国内运营商级 NAT，一般你的光猫上能获取 100.x.x.x 这样的 IP），那么通过一些复杂的奇技淫巧（例如 UDP hole punching）也能实现可被连接。
最差的情况是你的网络是 NAT4（对称形 NAT）的，这种情况下，你的网络是不可被连接的，除非你使用一些中继服务器来帮助你穿透。不过这种情况下，因为中继服务器的带宽有限，通常不能达到高速做种的要求，这里不讨论。

如何判断你是 NAT1 还是 NAT4 ？你可以使用 pystun3。

如果你使用 IPv6 做种（常见于教育网），运营商（学校）下发的 IPv6 地址一般都是公网 IP，不需要做任何设置，等同于情况一。

如果你不确定自己的网络是否可被连接，可以阅读文末注意点。

Peering

在数 Gbps 的高速网络下，你所在的运营商网络和你的种子的 peer 之间的连通性也值得考虑（但没那么重要）。运营商各张网之间，例如电信的 163 骨干网，教育网 CERNET ，甚至国外的运营商等，他们之间互联需要走 BGP ，如果你和你需要连接的用户之间连通性不好，那么你的上传速度可能会受到影响。

国内一般不需要考虑，这种一般在跨国传输的情况下比较明显，如果你所在的网的国际出口带宽不够，连接国外的 peer 时你会发现你的上传速度上不去。

足够快的硬盘

重要性仅次于网络的是硬盘速度，尤其关注 随机读。一种常见的错误认知是：我的机械硬盘顺序读能有 200MiB/s 那么我就能吃满 1Gbps 的上传带宽应该轻轻松松吧。这是完全错误的，由于做种会产生大量的 随机读 ，机械硬盘在这方面是最弱鸡的。

例如，在 1Gbps 左右上传，大文件做种（活跃数据超过我的内存大小 128GB ）的情况下，能稳定 10k IOPS 直达我的磁盘的读取（例如下图 14k IOPS 的读）。一块 7200 RPM 的 SATA 机械硬盘也就 100 IOPS ，一块 15000RPM 的 SAS 机械硬盘也就 200 IOPS 左右，这有着快 2 个数量级的差距。所以，如果你的硬盘是机械硬盘（HDD, spinning rust :p），建议放弃治疗，一旦内存缓存不够了你的上传速度就 gg 了，这辈子都不可能吃满 1Gbps 的，500Mbps 都够呛。根据我的经验， 3*8TB 的 7200RPM 企业级 HDD RAID5 在活跃数据超过内存缓存（128GB）的情况下也就勉勉强强能吃 200Mbps 的上传带宽（此时磁盘已经 100% busy 了），如果缓存更小的话这个数值甚至更低。

注意：这里不考虑完全使用内存缓存（因为大部分人都没有几百 GB 的内存）一旦种子活跃数据超过可用的内存缓存大小，硬盘就是你的上传速度的瓶颈。

所以高速做种的场景下一定要用 SSD ，最次也需要是 SATA SSD，建议使用更高性能的 NVME SSD 。

本文使用了 3 块 Micron 5400 PRO 1.92TB 的 SSD 组成的 RAID5 。注意，这对于我的网络接入（10 Gbps）的情况下仍然是远远不够的，正常需要高性能 NVME SSD 才行。不过我大部分情况也就 1Gbps 左右，跑不满 10Gbps ，凑合用吧。

如果你不确定你的硬盘是否足够快，那么你可以阅读下文判断磁盘 IO 瓶颈的部分。

足够大的内存

足够大的内存可以提供：

更大的磁盘缓存，减少磁盘压力（高速上传时，SSD 也不一定能跟上）；
更大的 send buffer ，提高网络传输效率/IO 吞吐；

一般来说 32GB 左右能够满足需求，本文中使用了 128GB 内存测试。

足够快的网卡

更好的网卡能够提供更多的 hardware offloading ，减轻 CPU 的负担，提高网络传输效率。一些高端网卡也支持更多的 queue ，提高并发传输能力，例如 CX-4 Lx 支持 64 个 queue ，可以看到这些 queue 上的收发包表示它们在被积极利用（实际上一般软件无法同时用不到这么多，包括 qBittorrent，所以 8 队列的 I350 就够了）。

最差也要使用 Intel 的 Gigabit 网卡，例如 I350 、I210 ，更好的可以用 Intel X520/X540 ，甚至 Mellanox ConnectX-4/5/6 等更高端的网卡。不要用 Realtek （小螃蟹）的，在大连接数吃满 1Gbps 的情况下性能很烂。

例如下图是我的计算机在 700Mbps 下行 1.2Gbps 上行时每秒的收发包数量，约为 170 Kpps 。性能差的网卡是很难达到这个性能的。

本文中使用了 Mellanox ConnectX-4 Lx 25G (CX4121A) 网卡（虽然现在被 NVIDIA 收购了，不过我还是习惯称之为 Mellanox ）。

足够快的 CPU

一个正常的现代 CPU 即可，例如 Intel Core i5-12500 。请不要使用树莓派、老掉牙的 CPU（例如 Intel Xeon E5-2403 v2 这种 4C4T 1.8GHz 的垃圾），这种情况下，你的 CPU 仍然是整个系统的瓶颈，特别是在高速上传时，CPU 的负载会很高。网卡越烂，上传速度越快，你就需要越好的 CPU 来处理。

本文中使用了国产的飞腾 2000+ ，虽然看起来有 64 个 ARM64 核心，但是性能还是比较烂的，不太够说实话，用起来跟只有 10 个核的 i9-10900K 也差的比较多。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
Address sizes: 44 bits physical, 48 bits virtual
CPU(s): 64
On-line CPU(s) list: 0-63
Thread(s) per core: 1
Core(s) per socket: 64
Socket(s): 1
NUMA node(s): 8
Vendor ID: Phytium
BIOS Vendor ID: Phytium
Model: 2
Model name: FTC662
BIOS Model name: FT-2000+/64
Stepping: 0x1
BogoMIPS: 100.00
L1d cache: 2 MiB
L1i cache: 2 MiB
L2 cache: 512 MiB
NUMA node0 CPU(s): 0-7
NUMA node1 CPU(s): 8-15
NUMA node2 CPU(s): 16-23
NUMA node3 CPU(s): 24-31
NUMA node4 CPU(s): 32-39
NUMA node5 CPU(s): 40-47
NUMA node6 CPU(s): 48-55
NUMA node7 CPU(s): 56-63
Flags: fp asimd evtstrm crc32 cpuid

软件调优

qBittorrent

基本设置

提升连接限制。如果你的网络/磁盘足够强劲，你可以去除所有的连接限制：

提高队列。磁盘性能足够的情况下，可以同时让多个种子做种，你也可以直接关闭：

提高内存限制。虽然目前（2024 年 6 月）这个设置只对 Windows 有效，不过设置了也无妨：

libtorrent 调优

我们针对 libtorrent >=2.0 版本，1 Gbps 左右的对等带宽， 64G 左右内存的用户，其他用户请酌情调整以下参数。

aio_threads Asynchronous I/O threads 默认 10 。例如在高速下载时，由于需要把下载的数据都坐 SHA1/SHA256 校验，CPU 可能存在瓶颈。提高这个值可以减轻让多核 CPU 并行处理。常见的说法是设置为 4 *线程数。由于我使用了 64 核的处理器，256 显得过大了，所以我设置为 64 。
file_pool_size File pool size 默认 40。一个 session 直接同时打开多少个文件。大量做种时，这个值设置的太小会导致频繁的文件打开/关闭，降低性能。而又设置的太大会超过一个进程的文件描述符限制（ulimit -n），导致打开文件失败。我设置为 900 （Linux 默认的文件描述符限制是 1024），留一些给其他的文件使用。
max_queued_disk_bytes Disk queue size 默认 1024 KiB 。这个值表示磁盘 IO 写入队列的大小，如果达到了这个上限，libtorrent 就会停止下载，直到磁盘完成了一部分写入。这个值设置的太小会导致下载性能下降，设置的太大会导致内存占用过高。我设置为 262144 KiB 。
default-disk-io-constructor Disk IO type 默认 mmap_disk_io_constructor 。这个值表示 libtorrent 使用的磁盘 IO 类型，posix_disk_io_constructor 是使用 POSIX IO ，mmap 是使用 mmap ，mmap 在 kernel 里有大量优化，建议只用 mmap （Memory Mapped Files）。如果你看到了大量内存占用，请看文末注意点。
disk_io_write_mode 和 disk_io_read_mode 默认 enable_os_cache 。维持默认。
piece_extent_affinity Use piece extent affinity 默认 0 。这个值表示 libtorrent 会尽量下载相邻的 piece ，提高磁盘 IO 吞吐。建议打开。
suggest_mode Send upload piece suggestions 默认 no_piece_suggestions 。这个值表示会告诉从我这边下载的 peer 建议下载那些在我读缓存里的 piece 而不是其他的没在缓存里的，可以提高做种速度减少磁盘 IO 。建议打开。
send_buffer_low_watermark Send buffer low watermark 默认 10 KiB。最小的目标 send buffer 大小（包括等待磁盘读取的字节数）。这实际上是初始窗口大小，它决定了我们能够多快地提高发送速率。我设置为 4096 KiB。
send_buffer_watermark Send buffer watermark 默认 500 KiB。如果发送缓冲区的字节数少于它，将从磁盘额外读取 16 KiB 的数据到缓冲区。这是上限，所以实际大小可能小于它。过小影响上传速率，过大浪费内存。我设置为 32768 KiB。
send_buffer_watermark_factor Send buffer watermark factor 默认 50。这是个百分比，peer 的上传速率乘以这个因子以获得实际的 send_buffer_watermark，但是不超过上面的 send_buffer_watermark。对于高速连接，设置得更高可以提高上传性能和磁盘吞吐，设置得过高可能会浪费内存并且偏好读。我的目的是高速做种，我设置为 200。
connection_speed Outgoing connections per second 默认 30。这个值表示每秒最多建立多少个出站连接，影响去下载 peer 那边数据的效率。我设置为 500 。
listen_queue_size Socket backlog size 默认 5 。它是传递给监听 socket listen() 的值，用于指定未处理传入连接的队列长度。当我们不积极等待连接被接受时，这些连接将会被排队。由于我们是大带宽高性能做种，我设置为 3000。
mixed_mode_algorithm μTP-TCP mixed mode algorithm 默认 prefer_tcp 。这个值用于确定在同时存在 TCP 和 uTP 连接时如何处理这两种连接。我们不需要为 uTP 做出让步，所以设置为 prefer_tcp 。
allow_multiple_connections_per_ip Allow multiple connections from the same IP address 默认 true 。这个值表示是否允许同一个 IP 建立多个连接。我们不需要限制，所以设置为 true 。
choking_algorithm Upload slots behavior 使用 fixed_slots_choker。
seed_choking_algorithm Upload choking algorithm 使用 fastest_upload。
max_out_request_queue Maximum outstanding requests to a single peer 默认 500 。表示发给 peer 的最大未处理请求数。我设置为 2000 。

自动下载/自动 Announce

一般种子流量最大的时候为刚发布之后，为了在种子发布后能够快速下载，抢占最早的上传机会。我们需要自动下载种子，然后立即开始做种。称之为 racing 。

这里我使用了自己编写的脚本，你也可以用 autodl-irssi 等工具。

注意点

判断磁盘 IO 瓶颈

使用 iostat -xm 1，找到存放种子数据的盘，特别关注最后的百分比，接近 100% 表示你的磁盘是瓶颈了。

你也可以在 htop 中看到 IO 的情况（在 Setup 的 Meter 中添加 Disk IO ）。htop 中的百分比是所有磁盘的合计，如果的下载盘是软 RAID 你有可能看到超过 100% 的情况，我这里是单个设备（硬 RAID），所以图中 99.1% 表示 IO 已经满了，我的硬盘已经跟不上了。

上面两张图是我使用的 3 块 SSD 阵列在 1~2 Gbps 上传，IO 爆发时的情况，可以看到其实 IO 已经快满了。如果你发现你的 busy% 接近 100% 了，那么你的磁盘 IO 就是瓶颈了，建议更换更好更快的 SSD。

如果你用的是机械硬盘，应该是轻轻松松全程 100% ，完全撑不住 :p

判断网络 IO 瓶颈

首先查看你的上传/下载速度是否超过了你的互联网带宽/物理网络接口的速度。你可以在 htop 中看到（记得在 Setup -> Meters 中添加 Network IO ）如果超过了，那么没什么办法，除非换网络/换网卡。如果没有，可以继续往下看。

如果你看到 softirq 占用过高，那么你的网络 IO 可能是瓶颈了，建议更换更好的 CPU 或网卡。

例如你可以在 htop 中看到紫色的 softirq 的占用（注意在 Setup -> Display Options 中勾选 Detailed CPU time ）。

如果你发现某个/几个核心的 softirq 占满了，那么你的网络 IO 可能是瓶颈了。

正常情况你不应该看到很高的 softirq 占用。

判断 CPU 瓶颈

一般来说，通过看 htop 即可。注意，你不仅要看全部的 CPU 核心占用，还要看单个核心的占用。如果你发现某个核占满了（这种情况更易发生）、整体 CPU 占用很高（目前 CPU 核心很多了，这个不太容易发生），那么你的 CPU 可能是瓶颈了。同时，看到 CPU 占用很高时，注意上面一节提到的 softirq 占用，这可能是网络 IO 的瓶颈。

例如在我的计算机上，我偶尔在 IO 爆发时能见到 CPU 占用 5000%+ 的峰值，这说明我的CPU对我的使用场景来说不太行。（图中其实你会发现我的磁盘IO也满了。。）

qBittorrent 内存占用过大？

如果你发现 qBittorrent 的内存占用过大，你不应该担心。这是 mmap 的特性，它会把文件映射到内存中，这样可以减少磁盘 IO ，提高性能。如果你关注总的内存占用，你会发现它并不会增加。自从 libtorrnet 2.0 使用 mmap 以来，有很多用户反馈内存占用过大，但是这是正常的。

例如我的计算机上，qBittorrent 的内存占用超过了 80GB ，但是总的内存占用只有 5GB。

检测是否可被连接

你可以使用外网 TCPing 来检查，例如一些网站可以替你检测： IPv4、IPv6 。输入你的公网 IP 和端口，如果各地均能够 ping 通，那么你是可被连接的。

上传仍然不及预期？

如果你各方面都确认没有瓶颈（网络、磁盘、CPU 等）但是上传速度仍然很慢，那么可能是你的种子不够热门，或者说你的 peer 的下行太慢了。这方面除了选择热门的种子（做种人少，下载人多，提高下载的 peer 连接你的机会），其实也没什么办法了。

因为你的上传是需要 peer 来连接你的，这就不是你能决定的事，可遇而不可求。举个例子，如果想跑满 10Gbps 的上传，那么需要 10 个 peer ，每个都以 1Gbps 的速度从你这拉数据，这是非常困难的。首先，别人的下载速度很少有 1G 的；其次，做种的人几十个，别人不可能只从你这下载，很可能从几十个不同的 peer 那边下载，一下就分散了，一个 peer 到你这只有几十分之一的流量了。

所以，这种情况下，除了选择合适的种子提高 peer 连接你的机会也没什么办法了。合适的种子通常是刚发布、热门、做种人少、下载人多的种子。

gVisor Killed My Node - A Debug Process

Mon, 15 Apr 2024 23:38:00 +0800

Recent Update: One month later, after we opened our issue #10264, it turns out that the gVisor developers are also aware of tha lacking of documentation about the systemd-cgroup option. Finally:

They have added the systemd-cgroup option to the documentation in 8c3abba, although the documentation website doesn’t seem to be updated as of now.

runsc will throw a warning if it detects a systemd-like path but systemd-cgroup is not used to tell you about the possible misconfiguration.

So if you are using gVisor with systemd cgroup, you should add the systemd-cgroup option to the runsc configuration, if not already.

The Problem

Recently we found some of our Kubernetes nodes were constant going down, completely dead, with no luck connecting to it. Taints like node.kubernetes.io/unreachable:NoSchedule and node.kubernetes.io/unreachable:NoExecute were automatically added to the nodes because the kubelet was not able to communicate with the API server. The only way to bring the node back was to restart it. After some debugging, we found out that some bad gVisor-created (runsc runtime) Pods was eating all the memory of the node and killing it. Traditional Pods with runc runtime were running fine.

You may say, “But there are memory limit on a Pod. If it uses more than the limit it should be killed by the OOM killer, right”？ That’s what I thought too, so I applied the following Deployment to intentionally consume some memory and see if it will be killed.

Click to see full yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
 app: memory-eater-bash
 name: memory-eater-bash
 namespace: default
spec:
 replicas: 1
 selector:
 matchLabels:
 app: memory-eater-bash
 template:
 metadata:
 labels:
 app: memory-eater-bash
 spec:
 containers:
 - command:
 - bash
 args:
 - -c
 - big_var=data; while true; do big_var="$big_var$big_var"; done; sleep 2d
 image: python:3.12-bookworm
 name: ubuntu
 securityContext:
 readOnlyRootFilesystem: true
 runAsNonRoot: true
 runAsUser: 999
 resources:
 limits:
 cpu: 100m
 memory: 256Mi
 requests:
 cpu: 100m
 memory: 200Mi
 dnsPolicy: Default
 hostNetwork: true
 restartPolicy: Always
 runtimeClassName: gvisor

I applied 256MiB memory limit on it, hoping it will be kill after it uses more than 256MiB memory. But it didn’t. It kept eating all the memory until the node was dead, bypass OOM protection provided by Kubernetes. It ate over 500GiB of memory, which is the total memory of the node, and we were no longer able to connect to the node. Luckily, we eviction process kicked in and existing Pods on the node are moved to other nodes, so the service was not affected by much.

The Debug Process

Of course, we need to apply the above memory-eater Deployment to reproduce the issue. But killing a node every time is not a good idea 😂 , especially when we need to manually restart it. So I used another Deployment to consume some (1GiB) memory, but not all of it.

Click to see full yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
 app: memory-eater-python
 name: memory-eater-python
 namespace: default
spec:
 replicas: 1
 selector:
 matchLabels:
 app: memory-eater-python
 template:
 metadata:
 labels:
 app: memory-eater-python
 spec:
 containers:
 - command:
 - python
 args:
 - -c
 - big_list = []; print('Allocating 1GB of memory...'); [big_list.append(' ' * 10**6) for _ in range(1000)]; import time; time.sleep(100000)
 image: python:3.12-bookworm
 name: ubuntu
 securityContext:
 readOnlyRootFilesystem: true
 runAsNonRoot: true
 runAsUser: 999
 resources:
 limits:
 cpu: 100m
 memory: 256Mi
 requests:
 cpu: 100m
 memory: 200Mi
 dnsPolicy: Default
 hostNetwork: true
 restartPolicy: Always
 runtimeClassName: gvisor

Just as I expected, the memory limit did not work. Time to debug.

Trying to reproduce

Before I trying to do anything on the production cluster, I tried to reproduce the issue on my local machine. Just to be safe not to kill the production cluster and get fired :P (just kidding. It’s because debugging is a lot easier on my local machine).

I used Docker with gVisor as per the official documentation. It was working fine. Our production cluster is using containerd, so was it a containerd issue? I was in a hurry so I used minikube with gVisor addon enabled, which should provide a Kubernetes cluster with containerd and gVisor (and a containerd shim for runsc). At the time of writing, it was Kubernetes v1.28.3, containerd v1.6.24, and gVisor release-20240401.0.

I know, I know. We should keep versions of the software same as the production cluster. But I was in a hurry and I was not sure if the issue was with containerd or gVisor. So I just went with it. Spoiler: yes, it has something to do with versions. If I were to do it again, I would use the same versions as the production cluster.

1

minikube start --addons="gvisor" --container-runtime=containerd --driver=docker --cpus=6

I applied the memory-eater Deployment and waited some surprising results. But the 256MiB memory limit was working as expected. The Pod was killed after it used more than 256MiB memory. I smelled some fishy things going on in the production cluster.

Debugging on the production cluster

Again, I applied the memory-eater Deployment, but on the production cluster this time. Hoping it will not kill the node. To my relief, it did not kill the node. But it did not kill the Pod with its memory (969MiB) above limits (256MiB) either (which is expected).

1
2
3


$ kubectl top pod memory-eater-python-67cc4dd4f7-sp2r6
NAME CPU(cores) MEMORY(bytes)
memory-eater-python-67cc4dd4f7-sp2r6 0m 969Mi

Before I dig anything deeper (like with some gVisor debugging tools), I checked the sandbox created by runsc (gVisor) to see if there is anything suspicious.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


# On production node
$ PID=$(ps aux | grep memory-eater-python | grep runsc-sandbox | awk '{print $2}')
$ cat /proc/$PID/cmdline | tr '\0' '\n'
runsc-sandbox
--log-format=json
--panic-log=/var/log/pods/default_memory-eater-python-74975bf59f-7fflh_1cf55082-d769-4a51-a424-2312c23eae82/gvisor_panic.log
--root=/run/containerd/runsc/k8s.io
--log=/run/containerd/io.containerd.runtime.v2.task/k8s.io/6a541285fa39d743dc20bd8b3169c4a8f5cd4d77542144fe82d3f6f7af469c2c/log.json
--log-fd=3
--panic-log-fd=4
boot
--apply-caps=false
--bundle=/run/containerd/io.containerd.runtime.v2.task/k8s.io/6a541285fa39d743dc20bd8b3169c4a8f5cd4d77542144fe82d3f6f7af469c2c
--controller-fd=10
--cpu-num=96
--io-fds=5,6
--mounts-fd=7
--overlay-mediums=0,0
--setup-root=false
--spec-fd=11
--start-sync-fd=8
--stdio-fds=12,13,14
--total-host-memory=540886331392
--total-memory=540886331392 # 503GiB
--user-log-fd=9
--product-name=AS
-4124GS-TNR
--proc-mount-sync-fd=22
6a541285fa39d743dc20bd8b3169c4a8f5cd4d77542144fe82d3f6f7af469c2c

Hmm, something caught my eye.

1
2


--total-host-memory=540886331392
--total-memory=540886331392 # 503GiB

Why is the total-memory and total-host-memory set to 503GiB? That’s the total memory of the node. I assume it should be set to the memory limit of the Pod, which is 256MiB. To verify my assumption, I checked the sandbox created by runsc on my local machine and it indeed was set to 256MiB.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


# On my local machine
$ PID=$(ps aux | grep memory-eater-python | grep runsc-sandbox | awk '{print $2}')
$ cat /proc/$PID/cmdline | tr '\0' '\n'
runsc-sandbox
--root=/run/containerd/runsc/k8s.io
--log=/run/containerd/io.containerd.runtime.v2.task/k8s.io/f11a26742777348875e6c7f182d9f8cc5c4a1c3a7f726ad5e8cb956b0c2dea96/log.json
--log-format=json
--panic-log=/var/log/pods/default_memory-eater-python-fc75975c8-vncwz_480a4dd9-22ad-4e43-9ecb-b797735e990a/gvisor_panic.log
--log-fd=3
--panic-log-fd=4
boot
--apply-caps=false
--bundle=/run/containerd/io.containerd.runtime.v2.task/k8s.io/f11a26742777348875e6c7f182d9f8cc5c4a1c3a7f726ad5e8cb956b0c2dea96
--controller-fd=10
--cpu-num=6
--dev-io-fd=-1
--gofer-mount-confs=lisafs:none,lisafs:none
--io-fds=5,6
--mounts-fd=7
--setup-root=false
--spec-fd=11
--start-sync-fd=8
--stdio-fds=12,13,14
--total-host-memory=2061119488
--total-memory=268435456 # 256MiB
--user-log-fd=9
--proc-mount-sync-fd=22
f11a26742777348875e6c7f182d9f8cc5c4a1c3a7f726ad5e8cb956b0c2dea96

It has the correct limits on my local machine.

1
2


--total-host-memory=2061119488
--total-memory=268435456 # 256MiB

So the issue was with this --total-memory argument. But why is it incorrect on the production cluster?

Checking the versions

Seeing the difference, I checked the versions on my local cluster and the production cluster.

Tool	Local	Production
Kubernetes	1.28.3	1.26.2
containerd	v1.6.24	v1.7.0-rc.1
runsc	20240401.0	20231009.0

Both are running Ubuntu Server 22.04 LTS, using systemd+cgroupsv2.

I assume the problem was with runsc (gVisor).

What’s wrong with gVisor?

Cgroups v2 support

After I know the issue is with the --total-memory flag. I found there is a similar issue reported google/gvisor #9580. The author had great explanation of the issue:

Under Kubernetes + cgroups v2 + systemd, gVisor launches all processes into the container subgroup associated with the pause container. This makes some sense given that cgroups v2 specifies that processes can only exist at leaf nodes, and the pod’s cgroup is registered as a slice (an intermediate unit which cannot have its own processes) with systemd. When the sandbox is launched gVisor needs a container subgroup and the pause container is the first to be launched. The pause container is a child of the pod cgroup and therefore inherits the limits of the parent pod cgroup, BUT the child’s controllers reflect the default max value. This in turn means that this code which reads the memory limit and cpu quota reads these as unlimited.

Let me explain a bit more. So a normal cgroup v2 hierarchy for a Pod looks like this:

1
2
3
4


/sys/fs/cgroup/kubepods.slice/
├── kubepods-burstable.slice
│ └── kubepods-burstable-pod<pod_id>.slice # has memory limit
│ └── cri-containerd-<container_id>.scope # no memory limit

The pod slice has a memory limit (cat memory.max gives xxx), but the container scope does not (cat memory.max gives max). The container scope is where the container is running. The container scope is a child of the pod slice. So the container should inherit the memory limit of the pod slice. This is how cgroups works.

But gVisor only checks the container scope and says “oh, it has no memory limit. Let’s set the total memory to max (the total memory of the node)”. And that’s why the --total-memory is set to 503GiB on the production cluster.

This issue was fixed in google/gvisor #9631. The code below is from the PR which fixed the issue (after the comments). So it checks the Pos slice (parent) if the memory limit is not set in the container slice (leaf node).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// File: runsc/cgroup/cgroup_v2.go
// Link: https://github.com/google/gvisor/blob/43c2c00c5006e7d21ce9cd431692622031891231/runsc/cgroup/cgroup_v2.go#L333-L346
// MemoryLimit returns the memory limit.
func (c *cgroupV2) MemoryLimit() (uint64, error) {
 limStr, err := getMemoryLimit(c.MakePath(""))
 if err != nil {
 return 0, err
 }
 // In cgroupv2+systemd, limits are set in the parent slice rather
 // than the leaf node. Check the parent to see if this is the case.
 if limStr == maxLimitStr {
 parentLimStr, err := getMemoryLimit(filepath.Dir(c.MakePath("")))
 if err != nil && !errors.Is(err, os.ErrNotExist) {
 return 0, err
 }
 if parentLimStr != "" {
 limStr = parentLimStr
 }
 if limStr == maxLimitStr {
 return math.MaxUint64, nil
 }
 }
 return strconv.ParseUint(limStr, 10, 64)
}

The runsc binary on our production cluster was built before the fix was merged. So I assume the issue was fixed in the runsc binary after the merge.

Systemd cgroup configuration

So if the issue was fixed in the latest release, simply upgrading the runsc binary should fix the issue. No. The issue was still there. I checked the runsc binary on the production cluster and it was built after the fix was merged. So what’s wrong?

Remember what a normal cgroup v2 hierarchy looks like in the previous section? Although the cgroups created by runc are just like that, that’s not how the ones created by runsc looks like on our production cluster. Instead, they look like this:

1
2
3
4
5
6


/sys/fs/cgroup/
├── kubepods.slice
│ ├── kubepods-burstable.slice
│ │ └── kubepods-burstable-pod<pod_id>.slice
├── system.slice
│ ├── kubepods-burstable-pod<pod_id>.slice:cri-containerd:<container_id>

Why are the container cgroups not under the pod cgroups? Why are they in system.slice? If that was the case, the fix above won’t do anything because the parent slice is not the pod slice. The parent slice is system.slice, it has no memory limit. So the container will have no memory limit even after the fix. OOM’in the node.

After some issue-searching on the gVisor GitHub repo, I found that runsc initially has no support for systemd cgroups google/gvisor #193. Support for systemd cgroup was added in google/gvisor #7287. It required a specific configuration in the runsc configuration:

1
2
3
4


// File: runsc/config/config.go
// Link: https://github.com/google/gvisor/blob/bf86207401cab99d859b25acd4911038608f0d33/runsc/config/config.go#L229-L230
+ // Use systemd to configure cgroups.
+ SystemdCgroup bool `flag:"systemd-cgroup"`

Hmm, looks like there was nothing like that in the runsc configuration on the production cluster. Go! Fix it! Find one of the cordoned node, drain it, upgrade the runsc binary, and change the configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


/etc/containerd/config.toml
 [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
 runtime_type = "io.containerd.runsc.v1"
+ [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc.options]
+ TypeUrl = "io.containerd.runsc.v1.options"
+ ConfigPath = "/etc/containerd/runsc.toml"

/etc/containerd/runsc.toml"
+ [runsc_config]
+ systemd-cgroup = "true"

Restart containerd systemctl restart containerd. Apply a new Pod with toleration and nodeSelector to schedule it to a cordoned node. And it worked! The container cgroups are now under the pod cgroups. The --total-memory is set to 256MiB. The Pod was killed after it used more than 256MiB memory. The node was not killed. The issue was fixed.

Conclusion

So the issue was a combination of old gVisor version and incorrect configuration. The gVisor version on the production cluster was built before the fix was merged. The configuration was missing the systemd-cgroup option. The issue was fixed by upgrading the runsc binary and adding the systemd-cgroup option to the configuration.

Thoughts on Re-Encoded BD Animes

Tue, 17 Jan 2023 21:28:00 +0800

Introduction

Re-encoded BD anime series are my go-to when watching animes. They have great quality while being a fraction of the size of a Blu-ray disc. Let alone they come with all kinds of extra contents.

Nowadays, re-encoded BD anime series usually have video tracks compressed using 10-bit x265 (a software implementation of the H.265/HEVC video compression standard), and audio tracks compressed using FLAC (a lossless audio codec).

The video tracks are great, generally a good balance between size and quality, no complaints. The audio tracks are my main problem. Sometimes they take up too much space with no real benefit for most viewers.

Other versions like WEB-DL are not considered because they are already heavily compressed so they don’t have the space concerns.

Video Tracks

I am totally satisfied with the video tracks, because:

the x265 encoder is very efficient (especially at variable bitrate controlling, which most hardware encoders still suffer from, i.e. NVIDIA NVENC, Intel QuickSync, and AMD VCN), being able to compress video to a fraction of the size of the BD original while preserving video quality;
the encoding team usually does a great job doing manual optimizations to fix problems in the original video, such as aliasing, ringing, and color banding. (Some team even goes as far as AI upscaling the video, although a bit controversial.)

Despite being one third (or even less) of the original bitrate, you can barely tell the difference between the re-encoded video and the original one, even when comparing side-by-side. In fact, the re-encoded video is sometimes even better due to the manual optimizations by the encoding team. This saves a lot of space while preserving video quality.

Video encoding is a complex topic, involving a huge amount of trial and error, even x265-encoding alone has a TON of dials and knobs to tweak, and this is only part of the whole encoding process. I will not go into details here, simply because of my lack of knowledge.

Samples

For example, you can see what the encoding team (VCB-Studio) does to optimize the original video quality from the note of Haiyore! Nyaruko-san / 潜行吧！奈亚子 :

For Season 1, the source is of nothing special. Its native resolution is 720p, then upscaled by a poor algorithm. This leads to many defects: lines featuring severe aliasing and ringing as well as serious ringing on the image borders. Dark scenes are widely spread throughout Season 1, together with the plentiful use of dark fades, resulting in heavy colour banding. For the lines, most of the problems can be solved by descaling and reconstruction, then slight AA and moderate de-ringing as supplements. For colour banding, since most scenes have only a little colour banding, in order to deal with severe colour banding in dark scenes, we designed a complex mask for protection in combination with luminance information. We also use a colour banding detection algorithm. Thanks to all those, we finally adaptively controlled the strength of de-banding. At the end, we added some grains to improve the viewing exprience.

第一季原盘画质一般。原生分辨率为 720p，被比较劣质的算法拉升上来，导致线条带有严重锯齿和振铃，而且画面的边框部分有很强的拉升带来的边缘振铃。画面暗场极多，而且大量运用暗式淡入淡出，这些场景出现了严重的色带。对于线条，大部分问题都可以通过逆向拉升再重构解决，然后补上轻微的抗锯齿处理和中等强度的去振铃处理。对于色带，由于大部分场景的色带都很少，只有少部分暗场有较严重的色带，我们结合亮度信息设计了复杂的保护手段，并使用了一个色带检测算法，根据检测的结果对去色带的力度进行了自适应调整。最后加上了一定强度的动态噪点，以改善原盘噪点的观感。

MediaInfo: Haiyore! Nyaruko-san episode 1 from VCB-Studio

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67


General
Unique ID : 276260962046491867529669363000855506406 (0xCFD5ED098E39DF78A31330DA2C4C2DE6)
Complete name : /Volumes/pool-raidz-3x4tb-0/public/anime/[VCB-Studio] Haiyore! Nyaruko-san/[VCB-Studio] Haiyore! Nyaruko-san [Ma10p_1080p]/[VCB-Studio] Haiyore! Nyaruko-san [01][Ma10p_1080p][x265_flac].mkv
Format : Matroska
Format version : Version 4
File size : 946 MiB
Duration : 23 min 56 s
Overall bit rate mode : Variable
Overall bit rate : 5 524 kb/s
Encoded date : UTC 2022-05-28 10:09:48
Writing application : mkvmerge v48.0.0 ('Fortress Around Your Heart') 64-bit
Writing library : libebml v1.4.0 + libmatroska v1.6.0

Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main 10@L4.1@High
Codec ID : V_MPEGH/ISO/HEVC
Duration : 23 min 56 s
Bit rate : 4 028 kb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 23.976 (24000/1001) FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 10 bits
Bits/(Pixel*Frame) : 0.081
Stream size : 690 MiB (73%)
Writing library : x265 3.5+97-ga456c6e73+1-g9859a8cb5:[Windows][clang 14.0.0][64 bit] Kyouko 10bit+8bit+12bit
Encoding settings : rc=crf / crf=14.0000 / qcomp=0.65 / qpstep=4 / stats-write=0 / stats-read=0 / vbv-maxrate=38000 / vbv-bufsize=40000 / vbv-init=0.9 / min-vbv-fullness=50.0 / max-vbv-fullness=80.0 / crf-max=0.0 / crf-min=0.0 / no-lossless / no-cu-lossless / aq-mode=3 / aq-strength=1.00 / aq-bias-strength=1.00 / cbqpoffs=-2 / crqpoffs=-2 / ipratio=1.40 / pbratio=1.20 / psy-rd=2.00 / psy-rdoq=1.00 / deblock=-1:-1 / ref=5 / limit-refs=0 / no-limit-modes / bframes=10 / b-adapt=2 / bframe-bias=0 / b-pyramid / b-intra / weightp / weightb / min-keyint=1 / max-keyint=360 / rc-lookahead=80 / gop-lookahead=0 / scenecut=40 / hist-scenecut=0 / radl=0 / max-cu-size=32 / min-cu-size=8 / me=3 / subme=5 / merange=38 / rdoq-level=1 / rd=5 / rdpenalty=0 / dynamic-rd=0.00 / rd-refine / ----- / cutree / no-sao / rect / no-amp / no-open-gop / wpp / no-pmode / no-pme / no-psnr / no-ssim / nr-intra=0 / nr-inter=0 / no-constrained-intra / no-strong-intra-smoothing / max-tu-size=16 / tu-inter-depth=4 / tu-intra-depth=4 / limit-tu=0 / qg-size=32 / qpmax=69 / qpmin=0 / ----- / cpuid=1111039 / frame-threads=4 / numa-pools=+ / log-level=2 / input-csp=1 / input-res=1920x1080 / interlace=0 / level-idc=0 / high-tier=1 / uhd-bd=0 / no-allow-non-conformance / no-repeat-headers / no-aud / no-hrd / info / hash=0 / no-temporal-layers / lookahead-slices=0 / no-splice / no-intra-refresh / no-ssim-rd / signhide / tskip / max-merge=5 / temporal-mvp / no-frame-dup / no-hme / no-analyze-src-pics / no-sao-non-deblock / selective-sao=0 / no-early-skip / no-rskip / no-fast-intra / no-tskip-fast / no-splitrd-skip / zone-count=0 / no-strict-cbr / no-rc-grain / no-const-vbv / sar=0 / overscan=0 / videoformat=5 / range=0 / colorprim=1 / transfer=1 / colormatrix=1 / chromaloc=0 / display-window=0 / cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / hist-threshold=0.03 / no-opt-cu-delta-qp / no-aq-motion / no-hdr10 / no-hdr10-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=0 / analysis-save-reuse-level=0 / analysis-load-reuse-level=0 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=1 / refine-ctu-distortion=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-analysis-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei / no-hevc-aq / no-svt / no-field / qp-adaptation-range=1.00 / scenecut-aware-qp=0conformance-window-offsets / right=0 / bottom=0 / decoder-max-rate=0 / no-vbv-live-multi-pass
Default : Yes
Forced : No
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709

Audio
ID : 2
Format : FLAC
Format/Info : Free Lossless Audio Codec
Codec ID : A_FLAC
Duration : 23 min 56 s
Bit rate mode : Variable
Bit rate : 1 494 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 11.719 FPS (4096 SPF)
Bit depth : 24 bits
Compression mode : Lossless
Stream size : 256 MiB (27%)
Writing library : libFLAC 1.3.2 (UTC 2017-01-01)
Language : Japanese
Default : Yes
Forced : No

Menu
00:00:00.000 : en:Chapter 01
00:01:14.992 : en:Chapter 02
00:02:45.040 : en:Chapter 03
00:13:08.997 : en:Chapter 04
00:22:10.037 : en:Chapter 05
00:23:40.002 : en:Chapter 06

As of video quality, you can verify yourself. Here are some frames extracted from Haiyore! Nyaruko-san / 潜行吧！奈亚子 (VCB-Studio). Can you tell any difference between the original and the re-encoded video? Remember that the re-encoded video has a bitrate of only 4028 kbps, while the original video will at least triple that.

This blog uses Responsive Images to improve experience, so what you see here may be scaled down. To view the original image, you can download the original file from the links provided.

Original	Re-encoded
original-0	re-encoded-0
original-1	re-encoded-1
original-2	re-encoded-2
original-3	re-encoded-3

Can’t notice any difference? Or the re-encoded one look better? Save the image locally and zoom in further :p. Yes, I DO put the correct pictures in the correct places.

The static images are already hard to tell apart. The difference will be even smaller when you are watching the video, simply because there are motions in the video and you will not be able to focus on the details.

So yeah, the video tracks are great. I will not touch them.

Audio Tracks

Lossless Compression Problem

However, the audio part is where I have a problem. It’s not about quality, but about size. Let me explain.

Unlike the video tracks, which is compressed using a lossy codec (H.265/HEVC), the audio tracks are usually losslessly compressed, which means that its quality is 100% same to the original, literally. This is great for archival purposes, but it’s a overkill for most people. Because what comes with great quality is size. FLAC tracks usually have a bitrate of 1000+ kbps. One FLAC track in a single episode will typically be over 200 MB in size, as you can seen from the example below. By compressing them to 200 kbps, for example, you save 800+ kbps per track, or put it simply, cut a 200 MB track to 40 MB, which saves a lot of space. This is especially true for animes with multiple FLAC tracks. An extreme example is Kobayashi-san Chi no Maidragon / Miss Kobayachi’s Dragon Maid / 小林家的龙女仆 from team AI-Raws. It has three FLAC tracks, with a total bitrate of 3955 (1432+1294+1229) kbps. Compressing them to 600 (200*3) kbps will cut the entire episode size from 1.89 GB to 1.3 GB, which is a 36% reduction in size, saving 400 MB per episode. That’s a lot.

MediaInfo: Kobayashi-san Chi no Maidragon episode 1 from AI-Raws

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109


General
Unique ID : 39274225615593709220200463866578723060 (0x1D8BF0D2BC4C9D643B9E040209C9E0F4)
Complete name : /Volumes/pool-raidz-3x4tb-0/public/anime/[AI-Raws][Miss Kobayachi's Dragon Maid][BDRip][MKV]/[AI-Raws] 小林さんちのメイドラゴン #01 (BD HEVC 1920x1080 yuv444p10le FLAC 日本語字幕)[6FDC51A9].mkv
Format : Matroska
Format version : Version 4
File size : 1.89 GiB
Duration : 25 min 11 s
Overall bit rate mode : Variable
Overall bit rate : 10.8 Mb/s
Encoded date : UTC 2022-01-24 14:17:11
Writing application : mkvmerge v33.1.0 ('Primrose') 64-bit
Writing library : libebml v1.3.7 + libmatroska v1.5.0

Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Format Range@L4@High
Codec ID : V_MPEGH/ISO/HEVC
Duration : 25 min 11 s
Bit rate : 6 700 kb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 23.976 (24000/1001) FPS
Chroma subsampling : 4:4:4
Bit depth : 10 bits
Bits/(Pixel*Frame) : 0.135
Stream size : 1.18 GiB (62%)
Writing library : x265 2.9+8-27d8424c799d:[Windows][MSVC 1900][64 bit] 10bit
Encoding settings : cpuid=1111039 / frame-threads=4 / numa-pools=16 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=3 / input-res=1920x1080 / interlace=0 / total-frames=0 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=4 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop / min-keyint=23 / keyint=250 / gop-lookahead=0 / bframes=4 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=25 / lookahead-slices=4 / scenecut=40 / radl=0 / no-intra-refresh / ctu=64 / min-cu-size=8 / rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=2 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / strong-intra-smoothing / max-merge=3 / limit-refs=3 / limit-modes / me=3 / subme=3 / merange=57 / temporal-mvp / weightp / no-weightb / no-analyze-src-pics / deblock=0:0 / sao / no-sao-non-deblock / rd=4 / no-early-skip / rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / no-b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=2.00 / psy-rdoq=1.00 / no-rd-refine / no-lossless / cbqpoffs=6 / crqpoffs=6 / rc=crf / crf=14.5 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=0 / vbv-maxrate=25600 / vbv-bufsize=10240 / vbv-init=0.9 / crf-max=0.0 / crf-min=0.0 / ipratio=1.40 / pbratio=1.30 / aq-mode=1 / aq-strength=1.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=31 / qpmin=0 / no-const-vbv / sar=0 / overscan=0 / videoformat=5 / range=0 / colorprim=1 / transfer=2 / colormatrix=2 / chromaloc=0 / display-window=0 / max-cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / no-hdr / no-hdr-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=5 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-mv-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei
Default : Yes
Forced : No
Color range : Limited
Color primaries : BT.709

Audio #1
ID : 2
Format : FLAC
Format/Info : Free Lossless Audio Codec
Codec ID : A_FLAC
Duration : 25 min 11 s
Bit rate mode : Variable
Bit rate : 1 432 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 11.719 FPS (4096 SPF)
Bit depth : 24 bits
Compression mode : Lossless
Stream size : 258 MiB (13%)
Writing library : libFLAC 1.2.1 (UTC 2007-09-17)
Default : Yes
Forced : No

Audio #2
ID : 3
Format : FLAC
Format/Info : Free Lossless Audio Codec
Codec ID : A_FLAC
Duration : 25 min 11 s
Bit rate mode : Variable
Bit rate : 1 294 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 11.719 FPS (4096 SPF)
Bit depth : 24 bits
Compression mode : Lossless
Stream size : 233 MiB (12%)
Title : キャストコメンタリー
Writing library : libFLAC 1.2.1 (UTC 2007-09-17)
Default : No
Forced : No

Audio #3
ID : 4
Format : FLAC
Format/Info : Free Lossless Audio Codec
Codec ID : A_FLAC
Duration : 25 min 11 s
Bit rate mode : Variable
Bit rate : 1 229 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 11.719 FPS (4096 SPF)
Bit depth : 24 bits
Compression mode : Lossless
Stream size : 221 MiB (11%)
Title : スタッフコメンタリー
Writing library : libFLAC 1.2.1 (UTC 2007-09-17)
Default : No
Forced : No

Text
ID : 5
Format : PGS
Muxing mode : zlib
Codec ID : S_HDMV/PGS
Codec ID/Info : Picture based subtitle format used on BDs/HD-DVDs
Duration : 24 min 30 s
Bit rate : 277 kb/s
Count of elements : 2364
Stream size : 48.5 MiB (3%)
Language : Japanese
Default : Yes
Forced : No

Lossy Compression

But compressing them so hard will certainly bring quality losses… Or will it? Well, yes, there must be quality loss when using lossy codecs, but the question is how much? Is it acceptable? You have seen that the example video tracks are compressed to 4028 kbps, which is a lot less than the original video. But the video tracks are still great. So, is the quality loss acceptable for audio tracks?

In fact, using a modern codec at 256 kbps (or even half that), the quality loss is so little that I can almost guarantee that you will NOT hear a difference for two reasons:

most codecs are considered “transparent” at 256 kbps or above (better codes will be “transparent” at even lower bitrate), which means that you cannot tell the difference between the original and the compressed one (more on this later);
when watching anime, you are usually not paying attention to details in the audio, but the video. This makes the difference in audio tracks even less noticeable.

So we can safely compress audio without worrying about quality loss. But how much should we compress? Well, that depends on your own preference. Generally, you can choose a codec and a bitrate that is better than what is “transparent” to you.

But hold on, how do I know what is “transparent” to me?

Audio Transparency

There is a concept called “transparent” in audio encoding, which means the compressed audio is so good that a person cannot tell a difference between the original and the compressed one. This is usually done through ABX blind tests, where the listener is not told which one is the original and which one is the compressed one. If the listener cannot tell the difference, then the audio is considered transparent. Of course, this differs from person to person. You have to do the test with your own ears with different encoders at different bitrates to find your “transparent” threshold (which codec at which bitrate). When you find your threshold, any better codec at any better bitrate will also be transparent to you.

About Bitrate and quality:

They cannot be compared directly. For example, 320 kbps MP3 is not the same as 320 kbps AAC. (AAC is generally considered a better codec than MP3, so it can achieve the same quality with a lower bitrate.)

When dealing with the same codec (same encoder, same settings), the higher the bitrate, the better the quality. For example, MP3 at 320 kbps wll have better quality than MP3 at 256 kbps.

But when dealing with different codecs, the bitrate is not the only factor that affects the quality. The codec itself, the encoder, encoding settings, and etc. all affect quality. For example, AAC (qaac, vbr, default settings) at 256 kbps is often said to have similar quality to MP3 at 320 kbps.

But the general consensus is that Opus is transparent at 160kbps and above, and AAC is transparent at 192kbps and above:

A blind test of multiple codecs at ~192kbps VBR shows that codecs at 192k is transparent to the test subject. Opus being the most transparent codec at 192k;
Tests of different encoders at different bitrates published by SoundExpert show that most codecs at 128kbps and above are transparent to most test subjects. Detailed results can be found at http://soundexpert.org/encoders-128-kbps;
A topic from HydrogenAudio forum even shows that Opus at ~80kbps is transparent to an average listener;

It’s interesting to observe that 4 members have mentioned Opus@80kbps as point where is hard to spot artifacts for them.

Opus ~80 kbps is roughly equivalent to LAME ~130 kbps (V5) which lands in an “excellent” area of quality (MOS 4.5+) http://listening-tests.hydrogenaud.io/sebastian/mp3-128-1/results.htm

So one could say that Opus 80 kbps is “excellent” at least for an average listener. It’s clear that an experienced listeners can spot artifacts at much higher rates.

I never noticed any artifacts at 80kbps though, though I haven’t tried to find any either. Quote from: noiselab on 2017-09-18 06:49:15

I tried it and for me it’s 80 kbit. Quote from: hlloyge on 2017-09-18 11:49:01

I’ll be honest, I struggled to ABX at 64kbps. 80kbps is enough for me. Quote from: Funkstar De Luxe on 2017-09-18 13:35:01

80kbps is pretty much my limit as well, can’t be bothered listening to killer samples all the time. Quote from: bstrobl on 2017-09-18 14:39:30
All iTunes music is delivered in AAC at 256kbps (before the arrival of Apple Music Lossless), which is an indication that most people cannot tell the difference as Apple is a company that cares about quality;

As you can see I am only focusing on AAC and Opus (as of lossy codecs), because AAC offers good sound quality and great compatibility while Opus has great sound quality and “okay” compatibility. Other codecs like MP3 are just not good enough to compete with them, so they will not be my choice later.

Listening Samples

Now bring your best audio equipment. Same as before, let’s listen to some samples to see if you can tell a difference, or you can do blind ABX tests by yourself to find your “transparent” threshold.

All lossy codecs are encoded using VBR (CBR is not considered because it is not size-efficient) with a quality setting to match the resulting bitrates of different encoders as close as possible.

Encoder versions

1
2
3
4


opusenc 0.2-3-gf5f571b; libopus 1.3.1
OggEnc v2.88; libvorbis 1.3.6
qaac 2.73; CoreAudioToolbox 7.10.9.0
lame 3.100

Note that if you cannot playback some audio tracks, make sure your browser supports this type of audio or you can download the audio tracks and play them locally. Latest versions of Chromium-based browsers should be fine. Safari might have some issues with Opus and Ogg.

Orange / オレンジ

For the first example, I will use Orange / オレンジ by 7!! — the 2nd ending song from Shigatsu wa Kimi no Uso / 四月は君の嘘 / Your Lie in April / 四月是你的谎言 for its female voices. The audio is not complex (simpley put, do not contain many instruments), so it is easier for the encoders to achieve good results.

Vocal

Reference track:

flac, best, 893k

Compressed tracks:

VBR Bitrate	Opus	AAC	Ogg	MP3
~256k	opusenc, vbr256, 276k	qaac, q109, 225k	oggenc2, q8, 234k	lame, v0, 249k
~192k	opusenc, vbr192, 212k	qaac, q91, 171k	oggenc2, q6, 178k	lame, v2, 182k
~160k	opusenc, vbr160, 180k	qaac, q82, 142k	oggenc2, q5, 153k	lame, v4, 148k
~128k	opusenc, vbr128, 146k	qaac, q64, 113k	oggenc2, q4, 131k	lame, v5, 126k
~96k	opusenc, vbr96, 111k	qaac, q45, 86k	oggenc2, q2, 92k	lame, v7, 98k
~64k	opusenc, vbr64, 76k	qaac, q27, 66k	oggenc2, q0, 61k	lame, v9, 69k

Vocal with simple instruments

Reference track:

flac, best, 920k

Compressed tracks:

VBR Bitrate	Opus	AAC	Ogg	MP3
~256k	opusenc, vbr256, 274k	qaac, q109, 236k	oggenc2, q8, 239k	lame, v0, 256k
~192k	opusenc, vbr192, 210k	qaac, q91, 180k	oggenc2, q6, 183k	lame, v2, 183k
~160k	opusenc, vbr160, 179k	qaac, q82, 149k	oggenc2, q5, 158k	lame, v4, 147k
~128k	opusenc, vbr128, 144k	qaac, q64, 119k	oggenc2, q4, 134k	lame, v5, 126k
~96k	opusenc, vbr96, 109k	qaac, q45, 90k	oggenc2, q2, 95k	lame, v7, 100k
~64k	opusenc, vbr64, 75k	qaac, q27, 66k	oggenc2, q0, 62k	lame, v9, 70k

Vocal with more complex instruments

Reference track:

flac, best, 1089k

Compressed tracks:

VBR Bitrate	Opus	AAC	Ogg	MP3
~256k	opusenc, vbr256, 262k	qaac, q109, 270k	oggenc2, q8, 266k	lame, v0, 288k
~192k	opusenc, vbr192, 200k	qaac, q91, 200k	oggenc2, q6, 201k	lame, v2, 206k
~160k	opusenc, vbr160, 169k	qaac, q82, 165k	oggenc2, q5, 170k	lame, v4, 154k
~128k	opusenc, vbr128, 136k	qaac, q64, 129k	oggenc2, q4, 142k	lame, v5, 134k
~96k	opusenc, vbr96, 102k	qaac, q45, 96k	oggenc2, q2, 97k	lame, v7, 101k
~64k	opusenc, vbr64, 69k	qaac, q27, 69k	oggenc2, q0, 62k	lame, v9, 68k

Vision

The second song will be a more complex, or demanding one — Vision by 中島岬:

Vision reference track:

flac, best, 1058k

Vision compressed tracks:

VBR Bitrate	Opus	AAC	Ogg	MP3
~256k	opusenc, vbr256, 260k	qaac, q109, 297k	oggenc2, q8, 290k	lame, v0, 288k
~192k	opusenc, vbr192, 197k	qaac, q91, 218k	oggenc2, q6, 212k	lame, v2, 209k
~160k	opusenc, vbr160, 165k	qaac, q82, 178k	oggenc2, q5, 175k	lame, v4, 156k
~128k	opusenc, vbr128, 133k	qaac, q64, 142k	oggenc2, q4, 139k	lame, v5, 135k
~96k	opusenc, vbr96, 100k	qaac, q45, 109k	oggenc2, q2, 99k	lame, v7, 109k
~64k	opusenc, vbr64, 66k	qaac, q27, 77k	oggenc2, q0, 63k	lame, v9, 73k

You can clearly see the AAC and MP3 encoders tend to give a higher bitrate than target bitrate in VBR mode to handle complex songs.

Want to listen to your own song at differnet bitrates? Use this script to encode it to all formats. This is exactly the same script that I wrote to encode the songs and generate the table above.

Now, you should have your own understanding of the different codecs: above which bitrate of which codec is transparent to you.

My take on the codecs

Here is my opinion: since ~192k Opus is already transparent to me (and to most people), why use 1000+ kbps FLAC? I can save a lot of space, without even tell a difference! Unless you have outstandingly gifted ears with high-end equipments, or you do heavy post-processing on audio, you should and will be fine with a transparent lossy encoding.

Also, most audio in animes is voice, which is usually not complex for encoders, so the actual bitrate can be even lower to sound good.

Still writing…

The rest is still being written…

Docker and ZFS - A Tough Pair

Sun, 07 Aug 2022 08:54:00 +0800

Outline:

Know ZFS filesystem and Docker’s zfs storage driver
How Docker’s zfs driver works and Why it is slow
Why we can’t directly use overlayfs on ZFS
- Why overlayfs don’t accept remote filesystems
- Why ZFS is identified as a remote filesystem
How to actually solve the problem

We will dive into the source code of moby, OpenZFS, and Linux kernel to find out.

Note: I admit this blog is not so beginner-friendly, which requires some prerequisites, otherwise you may have a hard time reading it though. I will give some questions or concepts after each prerequisite to help you know your understanding is enough on this topic.

general computer/unix concepts (block devices, copy-on-write, mount points)
basic filesystem concepts (difference between block devices and filesystems, common filesystems, basic understanding of Linux Virtual Filesystem)
the basics of ZFS filesystem (terminologies like datasets and snapshots, what are rollbacks)
Docker images (what are image layers, when they are created/deleted, how it works with UnionFS)

Background

What is ZFS

Described as The last word in filesystems, ZFS is scalable, and includes extensive protection against data corruption, support for high storage capacities, efficient data compression, integration of the concepts of filesystem and volume management, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, RAID-Z, native NFSv4 ACLs, and can be very precisely configured. ¹

By saying ZFS, I am referring to OpenZFS on Linux and FreeBSD: OpenZFS Documentation

ZFS is a great and sophisticated filesystem, really robust and stable. It never failed my expectations. I personally use ZFS on my personal devices, whenever possible, e.g. laptops (Ubuntu Desktop - for its built-in support for ZFS), NAS (TrueNAS SCALE), and servers (Proxmox VE and Ubuntu Server).

What is a Docker storage driver

Docker uses storage drivers to store image layers, and to store data in the writable layer of a container. The container’s writable layer does not persist after the container is deleted, but is suitable for storing ephemeral data that is generated at runtime. Storage drivers are optimized for space efficiency, but (depending on the storage driver) write speeds are lower than native file system performance, especially for storage drivers that use a copy-on-write filesystem. Write-intensive applications, such as database storage, are impacted by a performance overhead, particularly if pre-existing data exists in the read-only layer. ²

By default, Docker will use overlay2 whenever possible for all Linux distributions.

ZFS and Docker storage driver?

The Docker Engine provides a zfs storage drivers on Linux, which requires a ZFS filesystem, allowing for advanced options, such as creating snapshots, but require more maintenance and setup.

According to Docker docs, the zfs storage driver has the following advantages ³ :

Avoids the container’s writable layer grow too large in write-heavy workloads.
Performs better for write-heavy workloads (though not as well as Docker volumes).
A good choice for high-density workloads such as PaaS.

Hmm, sounds good, right? Well, keep reading. If it is that good, this blog wouldn’t exist at the first place.

In this blog, zfs refers to Docker’s zfs storage driver, mostly, but may also refer to ZFS filesystem. You should be able to distinguish them by context.

What’s the problem?

There is one single problem with ZFS that bothered me since the very beginning: Docker.

When using Docker on ZFS, one can only use its zfs driver (no, you cannot use overlay2 directly. We will see why later.). Although Docker docs proudly advertise its zfs driver as something high-performance:

zfs is a good choice for high-density workloads such as PaaS. ³

But in practice, this thing is slow as hell, specifically, when creating image layers. Build times can go from a fraction of a second on overlay2 to several minutes on zfs!

It is several magnitudes slower, a complete disaster.

Let’s take the Dockerfile in kube-trigger (a project that I worked on recently) as an example.

Click to see the complete Dockerfile

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59


# Copyright 2022 The KubeVela Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Note: the ${BIN} needs to be replaced with the actual binary,
# otherwise it won't work. Refer to Makefile for how it can be done.

ARG BUILD_IMAGE=golang:1.17
ARG BASE_IMAGE=gcr.io/distroless/static:nonroot

# Force native build platform, and cross-build to target platform later.
FROM --platform=${BUILDPLATFORM:-linux/amd64} ${BUILD_IMAGE} as builder

WORKDIR /workspace
COPY go.mod go.mod
COPY go.sum go.sum

ARG GOPROXY
ENV GOPROXY=${GOPROXY}
RUN go mod download

COPY build build
COPY hack hack
COPY cmd cmd
COPY api api
COPY controllers controllers
COPY pkg pkg

ARG TARGETARCH
ARG ARCH
ARG TARGETOS
ARG OS
ARG VERSION
ARG GOFLAGS
ARG DIRTY_BUILD
ARG ENTRY

RUN ARCH=${TARGETARCH:-${ARCH:-amd64}} \
 OS=${TARGETOS:-${OS:-linux}} \
 OUTPUT=${BIN} \
 VERSION=${VERSION} \
 GOFLAGS=${GOFLAGS} \
 /bin/sh build/build.sh \
 ${ENTRY}

FROM ${BASE_IMAGE}
WORKDIR /
COPY --from=builder /workspace/${BIN} .
ENTRYPOINT ["/${BIN}"]

We will focus on L39-L46:

1
2
3
4
5
6
7
8


ARG TARGETARCH
ARG ARCH
ARG TARGETOS
ARG OS
ARG VERSION
ARG GOFLAGS
ARG DIRTY_BUILD
ARG ENTRY

You might be thinking, this is just some build args, so what? Yes, this part almost does nothing (creates some image layers), and should finish immediately. That’s exactly the case on overlay2, but not on zfs, which will take minutes!

Such slow build times are driving me crazy.

Why `zfs` driver is so slow?

How ZFS storage driver works?

When using docker on a zfs dataset, the only option is Docker’s zfs driver, which uses ZFS dataset operations to create layered filesystems. The zfs storage driver for Docker stores each layer of each image as a separate legacy dataset. Even just a handful of images can result in a huge number of layers, each layer corresponding to a legacy ZFS dataset. As a result, there are hundreds of datasets created when only running a dozen containers.

The base layer of an image is a ZFS filesystem. Each child layer is a ZFS clone based on a ZFS snapshot of the layer below it. A container is a ZFS clone based on a ZFS Snapshot of the top layer of the image it’s created from. ⁴

Where’s the bottleneck?

Although when building images it do not have to deal with such many datasets. It will still spend a fair amount of time mounting and unmounting these datasets (can be seen from Docker debug logs).

We can take a look at the code from Docker daemon (moby/moby).

Mount will happen (if necessary) whenever Get is called:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


// File: daemon/graphdriver/zfs/zfs.go
// Link: https://github.com/moby/moby/blob/7e44b7cddd43b1771a44a2dd56548627e491c950/daemon/graphdriver/zfs/zfs.go#L365-L408
// Get returns the mountpoint for the given id after creating the target directories if necessary.
func (d *Driver) Get(id, mountLabel string) (_ containerfs.ContainerFS, retErr error) {
 d.locker.Lock(id)
 defer d.locker.Unlock(id)
 mountpoint := d.mountPath(id)
 if count := d.ctr.Increment(mountpoint); count > 1 {
 return containerfs.NewLocalContainerFS(mountpoint), nil
 }
 defer func() {
 if retErr != nil {
 if c := d.ctr.Decrement(mountpoint); c <= 0 {
 if mntErr := unix.Unmount(mountpoint, 0); mntErr != nil {
 logrus.WithField("storage-driver", "zfs").Errorf("Error unmounting %v: %v", mountpoint, mntErr)
 }
 if rmErr := unix.Rmdir(mountpoint); rmErr != nil && !os.IsNotExist(rmErr) {
 logrus.WithField("storage-driver", "zfs").Debugf("Failed to remove %s: %v", id, rmErr)
 }

 }
 }
 }()

 filesystem := d.zfsPath(id)
 options := label.FormatMountLabel("", mountLabel)
 logrus.WithField("storage-driver", "zfs").Debugf(`mount("%s", "%s", "%s")`, filesystem, mountpoint, options)

 root := d.idMap.RootPair()
 // Create the target directories if they don't exist
 if err := idtools.MkdirAllAndChown(mountpoint, 0755, root); err != nil {
 return nil, err
 }

 if err := mount.Mount(filesystem, mountpoint, "zfs", options); err != nil {
 return nil, errors.Wrap(err, "error creating zfs mount")
 }

 // this could be our first mount after creation of the filesystem, and the root dir may still have root
 // permissions instead of the remapped root uid:gid (if user namespaces are enabled):
 if err := root.Chown(mountpoint); err != nil {
 return nil, fmt.Errorf("error modifying zfs mountpoint (%s) directory ownership: %v", mountpoint, err)
 }

 return containerfs.NewLocalContainerFS(mountpoint), nil
}

Unmount will happen whenever Put is called:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


// File: daemon/graphdriver/zfs/zfs.go
// Link: https://github.com/moby/moby/blob/7e44b7cddd43b1771a44a2dd56548627e491c950/daemon/graphdriver/zfs/zfs.go#L410-L431
// Put removes the existing mountpoint for the given id if it exists.
func (d *Driver) Put(id string) error {
 d.locker.Lock(id)
 defer d.locker.Unlock(id)
 mountpoint := d.mountPath(id)
 if count := d.ctr.Decrement(mountpoint); count > 0 {
 return nil
 }

 logger := logrus.WithField("storage-driver", "zfs")

 logger.Debugf(`unmount("%s")`, mountpoint)

 if err := unix.Unmount(mountpoint, unix.MNT_DETACH); err != nil {
 logger.Warnf("Failed to unmount %s mount %s: %v", id, mountpoint, err)
 }
 if err := unix.Rmdir(mountpoint); err != nil && !os.IsNotExist(err) {
 logger.Debugf("Failed to remove %s mount point %s: %v", id, mountpoint, err)
 }

 return nil
}

Although Docker will not mount a filesystem twice, changes still exist when consecutive Get/Put call happens.

I am not an OpenZFS developer, but it seems to me that there is a bottleneck with ZFS with such frequent mount/unmount actions (with a large mount datasets and snapshots).

As you can see, Docker is already optimizing this situation by using the mount syscall directly (instead of calling user-space mount command, which will again, after going to kernel-space from user-space, require the kernel to call the zfs mount binary in user-space, due to ZFS’s license issues with the vfs_mount in kernel).

One possible solution

So there is not much to optimize in zfs storage drivers. It is the actual zfs mount process that is slowing image build times down. Now, the problem with Docker’s zfs storage driver is clear. There are two options left:

optimize ZFS mount times
just get rid of zfs storage driver

Of course, you can also grab another disk in ext4 as use overlayfs on top of it. But I only have zfs-formatted disks, so I only have the above two options.

The first one “optimize ZFS mount times” isn’t really an option. Currently, I don’t have the expertise or time to work on OpenZFS.

With that out of the way, we only have one option left: “do not use zfs storage driver”, i.e., use overlay2 on zfs datasets.

But it doesn’t work

Now, the problem is, how do we use overlay2 storage driver on a ZFS filesystem (dataset)?

Simply put, that’s not possible (directly). ZFS makes use of d_revalidate . Having d_revalidate set to not NULL will make overlayfs refuse to work.

But why? To understand, we need to analyze some source code from OpenZFS and Linux kernel.

What’s `d_revalidate`?

d_revalidate is defined in Linux kernel include/linux/dcache.h:

1
2
3
4
5
6


// File: include/linux/dcache.h
// Link: https://github.com/torvalds/linux/blob/1612c382ffbdf1f673caec76502b1c00e6d35363/include/linux/dcache.h#L128
struct dentry_operations {
 int (*d_revalidate)(struct dentry *, unsigned int);
 // The rest are omitted.
} ____cacheline_aligned;

The kernel documentation has a nice description on d_revalidate.

TL;DR: d_revalidate is typically used with network filesystems, and is called called when the VFS needs to revalidate a dentry, marking this dentry is still valid or not, to prevent things change without the client being aware of it.

d_revalidate is called called when the VFS needs to revalidate a dentry. This is called whenever a name look-up finds a dentry in the dcache. Most local filesystems leave this as NULL, because all their dentries in the dcache are valid. Network filesystems are different since things can change on the server without the client necessarily being aware of it.

This function should return a positive value if the dentry is still valid, and zero or a negative error code if it isn’t.

d_revalidate may be called in rcu-walk mode (flags & LOOKUP_RCU). If in rcu-walk mode, the filesystem must revalidate the dentry without blocking or storing to the dentry, d_parent and d_inode should not be used without care (because they can change and, in d_inode case, even become NULL under us).

If a situation is encountered that rcu-walk cannot handle, return -ECHILD and it will be called again in ref-walk mode.

Excerpt from: Overview of the Linux Virtual File System — The Linux Kernel documentation

Why use `d_revalidate`?

ZFS makes use of d_revalidate to invalidate dcache after rolling back.

See? This is the same situation as the kernel doc describes. When a roll back happens, the underlying files are changed, but the dcache it not updated, so we need to use d_revalidate to mark it as invalid.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


// File: module/os/linux/zfs/zpl_inode.c
// Link: https://github.com/openzfs/zfs/blob/1d3ba0bf01020f5459b1c28db3979129088924c0/module/os/linux/zfs/zpl_inode.c#L701-L739
static int
#ifdef HAVE_D_REVALIDATE_NAMEIDATA
zpl_revalidate(struct dentry *dentry, struct nameidata *nd)
{
 unsigned int flags = (nd ? nd->flags : 0);
#else
zpl_revalidate(struct dentry *dentry, unsigned int flags)
{
#endif /* HAVE_D_REVALIDATE_NAMEIDATA */
 /* CSTYLED */
 zfsvfs_t *zfsvfs = dentry->d_sb->s_fs_info;
 int error;

 if (flags & LOOKUP_RCU)
 return (-ECHILD);

 /*
 * After a rollback negative dentries created before the rollback
 * time must be invalidated. Otherwise they can obscure files which
 * are only present in the rolled back dataset.
 */
 if (dentry->d_inode == NULL) {
 spin_lock(&dentry->d_lock);
 error = time_before(dentry->d_time, zfsvfs->z_rollback_time);
 spin_unlock(&dentry->d_lock);

 if (error)
 return (0);
 }

 /*
 * The dentry may reference a stale inode if a mounted file system
 * was rolled back to a point in time where the object didn't exist.
 */
 if (dentry->d_inode && ITOZ(dentry->d_inode)->z_is_stale)
 return (0);

 return (1);
}

d_revalidate is set to the zpl_revalidate function that we have seen above.

1
2
3
4
5


// File: module/os/linux/zfs/zpl_inode.c
// Link: https://github.com/openzfs/zfs/blob/1d3ba0bf01020f5459b1c28db3979129088924c0/module/os/linux/zfs/zpl_inode.c#L830-L832
dentry_operations_t zpl_dentry_operations = {
 .d_revalidate = zpl_revalidate,
};

But why is having d_revalidate set to not NULL will make overlayfs refuse to work?

How `overlayfs` refuses `d_revalidate`-enabled fs

To understand how and why How overlayfs refuses d_revalidate-enabled fs, let’s turn our focus to the Linux kernel.

`DCACHE_OP_REVALIDATE` flag is set

If a dentry has d_revalidate set to not NULL, which is the case with ZFS, kernel will mark DCACHE_OP_REVALIDATE in its d_flags. A d_flags is just flags to tell what operations that this dentry supports, and DCACHE_OP_REVALIDATE means it supports d_revalidate operations.

Now in our case, ZFS uses d_revalidate, so our d_flags have a DCACHE_OP_REVALIDATE present.

Keep this in mind. This flag will cause overlayfs to identify it as a remote fs. You will see why later.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


// File: fs/dcache.c
// Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/dcache.c#L1915-L1943
void d_set_d_op(struct dentry *dentry, const struct dentry_operations *op)
{
 WARN_ON_ONCE(dentry->d_op);
 WARN_ON_ONCE(dentry->d_flags & (DCACHE_OP_HASH |
 DCACHE_OP_COMPARE |
 DCACHE_OP_REVALIDATE |
 DCACHE_OP_WEAK_REVALIDATE |
 DCACHE_OP_DELETE |
 DCACHE_OP_REAL));
 dentry->d_op = op;
 if (!op)
 return;
 if (op->d_hash)
 dentry->d_flags |= DCACHE_OP_HASH;
 if (op->d_compare)
 dentry->d_flags |= DCACHE_OP_COMPARE;
 if (op->d_revalidate)
 dentry->d_flags |= DCACHE_OP_REVALIDATE;
 if (op->d_weak_revalidate)
 dentry->d_flags |= DCACHE_OP_WEAK_REVALIDATE;
 if (op->d_delete)
 dentry->d_flags |= DCACHE_OP_DELETE;
 if (op->d_prune)
 dentry->d_flags |= DCACHE_OP_PRUNE;
 if (op->d_real)
 dentry->d_flags |= DCACHE_OP_REAL;

}
EXPORT_SYMBOL(d_set_d_op);

Mounting process of `overlayfs`

To understand how overlayfs rejects d_revalidate enabled fs, we need to look at the code that mounts overlayfs.

When we mount a overlayfs, we call ovl_mount() in the kernel fs/overlayfs.

1
2
3
4
5
6
7


// File: fs/overlayfs/super.c
// Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/overlayfs/super.c#L2158-L2162
static struct dentry *ovl_mount(struct file_system_type *fs_type, int flags,
 const char *dev_name, void *raw_data)
{
 return mount_nodev(fs_type, flags, raw_data, ovl_fill_super);
}

ovl_fill_super() will be called to fill the dir (workdir) where overlayfs mounts, which will call ovl_get_workdir.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


// File: fs/overlayfs/super.c
// Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/overlayfs/super.c#L2047-L2079
static int ovl_fill_super(struct super_block *sb, void *data, int silent)
{
 // Omitted till L2048
 if (ofs->config.upperdir) {
 struct super_block *upper_sb;

 err = -EINVAL;
 if (!ofs->config.workdir) {
 pr_err("missing 'workdir'\n");
 goto out_err;
 }

 err = ovl_get_upper(sb, ofs, &layers[0], &upperpath);
 if (err)
 goto out_err;

 upper_sb = ovl_upper_mnt(ofs)->mnt_sb;
 if (!ovl_should_sync(ofs)) {
 ofs->errseq = errseq_sample(&upper_sb->s_wb_err);
 if (errseq_check(&upper_sb->s_wb_err, ofs->errseq)) {
 err = -EIO;
 pr_err("Cannot mount volatile when upperdir has an unseen error. Sync upperdir fs to clear state.\n");
 goto out_err;
 }
 }

 err = ovl_get_workdir(sb, ofs, &upperpath);
 if (err)
 goto out_err;

 if (!ofs->workdir)
 sb->s_flags |= SB_RDONLY;

 sb->s_stack_depth = upper_sb->s_stack_depth;
 sb->s_time_gran = upper_sb->s_time_gran;
 }
}

Saw something called lowerdir and upperdir? Here’s what they means in overlayfs:

An overlay filesystem combines two filesystems - an ‘upper’ filesystem and a ’lower’ filesystem. When a name exists in both filesystems, the object in the ‘upper’ filesystem is visible while the object in the ’lower’ filesystem is either hidden or, in the case of directories, merged with the ‘upper’ object.

It would be more correct to refer to an upper and lower ‘directory tree’ rather than ‘filesystem’ as it is quite possible for both directory trees to be in the same filesystem and there is no requirement that the root of a filesystem be given for either upper or lower.

The lower filesystem can be any filesystem supported by Linux and does not need to be writable. The lower filesystem can even be another overlayfs. The upper filesystem will normally be writable and if it is it must support the creation of trusted.* extended attributes, and must provide valid d_type in readdir responses, so NFS is not suitable.

A read-only overlay of two read-only filesystems may use any filesystem type.

Excerpt from Overlay Filesystem — The Linux Kernel documentation

And ovl_get_workdir will get where the workdir is and it call ovl_make_workdir to make the workdir.

1
2
3
4
5
6
7
8


// File: fs/overlayfs/super.c
// Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/overlayfs/super.c#L1513
static int ovl_get_workdir(struct super_block *sb, struct ovl_fs *ofs,
 struct path *upperpath)
{
 // Omitted till L1513
 err = ovl_make_workdir(sb, ofs, &workpath);
}

ZFS is identified as a remote fs

Now the good bit comes, notice the comments. This where the overlayfs rejects remote fs. (We will see why ZFS is identified as remote fs later.)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


// File: fs/overlayfs/super.c
// Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/overlayfs/super.c#L1434-L1444
static int ovl_make_workdir(struct super_block *sb, struct ovl_fs *ofs,
 struct path *workpath)
{
 // Omitted till L1433
 /*
 * We allowed sub-optimal upper fs configuration and don't want to break
 * users over kernel upgrade, but we never allowed remote upper fs, so
 * we can enforce strict requirements for remote upper fs.
 */
 if (ovl_dentry_remote(ofs->workdir) &&
 (!d_type || !rename_whiteout || ofs->noxattr)) {
 pr_err("upper fs missing required features.\n");
 err = -EINVAL;
 goto out;
 }
}

In ovl_dentry_remote, it directly marks dentry which has DCACHE_OP_REVALIDATE flags (Remember what we said before? ZFS sets this flag.) as remote, and thus the code above will go into the if-condition, then rejecting it.

1
2
3
4
5
6
7


// File: fs/overlayfs/super.c
// Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/overlayfs/util.c#L97-L101
bool ovl_dentry_remote(struct dentry *dentry)
{
 return dentry->d_flags &
 (DCACHE_OP_REVALIDATE | DCACHE_OP_WEAK_REVALIDATE);
}

Now everything comes together. Ah, this is why having d_revalidate set to not NULL will lead to Linux treating ZFS as a remote filesystem (like NFS) and thus things like overlayfs won’t work with ZFS.

There is PRs in OpenZFS to fix this problem: https://github.com/openzfs/zfs/pull/9600 , https://github.com/openzfs/zfs/pull/9414 . But currently they are held and I don’t the expertise or time to work on it either. Hopefully wish I can pick up that PR and fix it (if possible).

Final solution

So, the only option left is not possible now. Is there something we can do?

As I said earlier, “Simply put, that’s not possible (directly)”. Well, it turns out, there is still an indirect way – ZFS Volumes. Let’s see what Oracle says:

A ZFS volume is a dataset that represents a block device.

Excerpt from: https://docs.oracle.com/cd/E19253-01/819-5461/gaypf/index.html

Note that it is a block device. This is really important, which means we can treat it like a conventional hard drive and do whatever we want on ZFS datasets!

Since it is a block device, we can use it as a Swap device, iSCSI target, and in this case, a block device holding a ext4 partitation to put overlayfs on.

Solve problem

Finally! We now decide to use ZFS Volumes (zvol) to hold our overlayfs , i.e., overlayfs on top of ext4 on top of zvol on top of ZFS datasets. (Well, it is a bit complex. But trust me, even with so many fs layers, the performance is still wayyyy higher than Docker’s zfs driver.)

Let’s fix this now.

Stop Docker:

1

sudo systemctl stop docker

Destroy the dataset that Docker uses previously. You can use zfs list to find all datasets. In our case, it is rpool/ROOT/ubuntu_uzcb39/var/lib/docker.

1
2
3


# I mount this dataset at /var/lib/docker, which docker uses. Remove it.
sudo zfs destroy rpool/ROOT/ubuntu_uzcb39/var/lib/docker -R -r
# Be careful! This will destroy datasets recursively.

Create a ZFS Volume.

1
2
3
4
5


sudo zfs create -sV 64G rpool/ROOT/ubuntu_uzcb39/var/lib/docker
# rpool/ROOT/ubuntu_uzcb39/var/lib/docker is where the dataset is. Note that this will not be mounted to /var/lib/docker, which is different from the one above.
# -V creates a zvol
# -s makes it sparse, i.e, dynamically expands instead of taking all defined space
# set a max size of 64G

Format the zvol. ZFS Volumes are identified as devices in the /dev/zvol/{dsk,rdsk}/pool directory. Since we created a block device, let’s format it to ext4.

1
2


# The zvol we just created is mounted at /dev/zvol/rpool/ROOT/ubuntu_uzcb39/var/lib/docker.
sudo mkfs.ext4 /dev/zvol/rpool/ROOT/ubuntu_uzcb39/var/lib/docker

Mount the ext4 partitation to /var/lib/docker.

1
2
3


sudo mkdir -p /var/lib/docker
# Mount ext4-formatted zvol (block device) to /var/lib/docker
sudo mount /dev/zvol/rpool/ROOT/ubuntu_uzcb39/var/lib/docker /var/lib/docker

Check if it is successfully mounted.

1
2
3


df -hT
# /dev/zd0 ext4 63G 5.8G 54G 10% /var/lib/docker
# It is mounted, great!

Start Docker back up and check status.

1
2
3
4
5
6
7
8


sudo systemctl start docker
docker info
# We are using overlay2 now!
# > Storage Driver: overlay2
# > Backing Filesystem: extfs
# > Supports d_type: true
# > Native Overlay Diff: true
# > userxattr: false

Make changes persistent. Make sure the zvol is automatically mounted to /var/lib/docker after system reboots.

1
2
3


sudo vim /etc/fstab
# Append this:
# /dev/zvol/rpool/ROOT/ubuntu_uzcb39/var/lib/docker /var/lib/docker ext4 defaults 0 0

Horrey! Build time are several orders of magnitude faster now!

Search

Mon, 01 Jan 0001 00:00:00 +0000

Charlie Chiang's blog

Building Alpine Linux Disk Images Without a VM

Why No VM Is Needed

The Disk Layout

The MBR Variant

What It Trusts

Configuration

Choosing A Root Filesystem

Tuning mkfs For The Storage Underneath

Tuning Mount Options

The Mount Option Gotcha

Btrfs Subvolumes

Compression Needs Forcing At Build Time

Sizing /boot

Output Formats

What Must Not Survive Cloning

Staying Able To Get In

Hooks

Writing Your Own

Verifying The Image

Does It Boot

Is It Configured Correctly

Everything At Once

Uploading

What I Kept From The Old Post

Building Audio Insight: An Audio Analyzer That Actually Renders Smoothly

What an audio plugin actually does

Turning samples into pictures

Spectrum: what frequencies exist now?

Spectrogram: how did the spectrum change?

Peak and RMS: how strong is the signal?

Stereo: how are left and right related?

Loudness: how loud does it feel over time?

Smooth is a timing property, not an FPS number

The plugin that crashed its host

Why 120 display callbacks produced 60 frames

Making 60 Hz data scroll on a 120 Hz display

The random resets that were not random

Building observability into the plugin

An agent-assisted, human-tested development loop

What is open source today

Installing Proxmox VE on an OVHcloud Kimsufi KS-5

Hardware

Why Install Proxmox Manually

Format NVMe Drives as 4K LBA

Open the IPMI KVM

Do Not Stream the Proxmox ISO Through JViewer

Boot Proxmox Through netboot.xyz

Install Proxmox on ZFS RAID1

Optional Partition Alignment Fix

Fix OVHcloud Boot-to-Disk

Basic Post-Install ZFS Settings

Reduce Unnecessary Host Writes

Configure BBR

Configure Swap

SSH and Serial Console

Scratch Storage

mdadm + LVM Alternative

Using Scratch Storage in VMs and CTs

TRIM

Proxmox Networking on a Single Public IP

NAT and Port Forwarding

DHCP and DNS with Pi-hole

Host Monitoring

Service CT

SMTP Notifications

Final State

Minimal Alpine Linux on a 1 GB Btrfs Root Disk

Assumptions

Partition The Disk

Start Alpine Setup

Format And Mount

Install Alpine

Verify fstab And initramfs

Fix The MBR Bootloader

Convert The Disk To QCOW2

Restore From The Provider’s Original OS

Restore Without A Console Or Rescue Mode

Things I Bake Into The Image

SSH

Sizing `/boot`