[{"content":"In Minimal Alpine Linux on a 1 GB Btrfs Root Disk I wrote down the recipe I use to build a tiny Alpine instance: boot the ISO in a VM, run setup-alpine with disk=none, partition by hand, setup-disk -m sys /mnt, reboot, then qemu-img convert the disk. It works, and I still use that post as the explanation of why the pieces are the way they are.\nWhat I got tired of is the process. Every image costs a VM, a console session, and a sequence of interactive answers, and none of it is reproducible: if I want the same image with ext4 instead of btrfs, I do the whole thing again and hope I remember every step.\nSo I replaced it with a script. It builds the disk image as a file, on any Linux host, with no VM and no prompts. It also fixes the two things I did not like about the old recipe: the image now boots under both BIOS and UEFI, and every choice is configurable.\nThe bundle for this post contains everything:\nmkalpine.sh — the builder config.example.sh — every knob with comments testboot.sh — boot the result under QEMU and check it came up testmatrix.sh — build and boot every supported combination hooks/ — the customizations, one file each 1 sudo ./mkalpine.sh -f alpine.img That is the whole interface. About a minute later there is a 512 MiB sparse image, 102 MiB of it actually allocated, that boots on either firmware. The image is deliberately small — 70-growroot expands the root filesystem to fill whatever disk it lands on, so IMAGE_SIZE only has to hold the build.\nWhy No VM Is Needed The manual recipe suggests that installing Alpine requires a running Alpine. It does not. Once you look at what the installer actually does, almost all of it is file manipulation:\nUnpack a root filesystem. apk add a kernel, an initramfs generator, and a bootloader. Write /etc/fstab, /etc/inittab, and the network config. Run mkinitfs and grub-install. Enable some OpenRC services, which is ln -s in a directory. Only steps 2 and 4 need to execute Alpine binaries, and a chroot is enough for that. There is no step that needs a booted kernel, a real block device, or firmware.\nAlpine\u0026rsquo;s own setup-disk cannot be reused here, unfortunately. It sources libalpine.sh, calls lbu package, and assumes throughout that it is running on a booted live system. So mkalpine.sh reimplements the same steps directly, following the structure of upstream alpine-make-vm-image and the bootloader logic from setup-disk.\nThe pipeline is:\n1 2 3 4 5 6 7 8 9 10 11 truncate -s 512M image create a sparse file sfdisk partition it losetup -P get /dev/loopNp1..p3 mkfs.vfat / mkfs.btrfs make filesystems mount mount root, then /boot inside it tar -x minirootfs unpack the base system chroot + apk add kernel, mkinitfs, grub, fs tools mkinitfs / grub-install bootloader for both firmwares hooks/ customizations fstrim; umount; losetup -d compact and release qemu-img / zstd / gzip optional output formats QEMU is not needed to build. It is used only for qcow2 output and by the boot tests.\nOne line of that pipeline is less innocent than it looks. losetup -P asks the kernel to scan the partition table, but the /dev/loopNp* nodes are created by udev, after losetup has already exited, so they have to be waited for — and on a host with no udev at all, such as a container whose /dev is a plain tmpfs, they have to be created by hand from the device numbers the kernel publishes in /sys/block/loopN/loopNpM/dev. Both paths are in the script, and the interesting part is the order they are tried in: reaching for mknod early is a mistake, because when udev gets round to the same events it unlinks your node and makes its own, and for the instant in between the path does not exist. On a test host, that instant landed exactly on mkfs.vfat:\n1 mkfs.vfat: unable to open /dev/loop0p2: No such file or directory So the escalation is now slow on purpose — udevadm settle, a second, partx -a, another second, and only then mknod — and once the nodes do appear there is one more settle before anything opens them, so a replacement in flight finishes before mkfs runs rather than during it.\nThe Disk Layout 1 2 3 4 5 6 7 8 9 10 GPT (default; the protective MBR still carries GRUB\u0026#39;s boot.img) # size type mount contents 1 1 MiB ef02 BIOS boot - grub core.img (x86_64 only) 2 64 MiB ef00 ESP FAT32 /boot vmlinuz-virt, initramfs-virt, grub/, EFI/BOOT/BOOTX64.EFI 3 rest L Linux / btrfs (default), ext4 or xfs BIOS : firmware -\u0026gt; MBR boot.img -\u0026gt; p1 core.img -\u0026gt; /boot/grub/grub.cfg UEFI : firmware -\u0026gt; p2 /EFI/BOOT/BOOTX64.EFI -\u0026gt; /boot/grub/grub.cfg Two details are worth calling out.\n/boot is the ESP. This is not just to save a partition. It means GRUB only ever has to read FAT. Btrfs, ext4 and XFS roots all work without GRUB parsing them at all, and no on-disk feature the root filesystem gains later can break the bootloader. The old post used Syslinux on ext4 /boot; this is less fragile.\nGRUB is installed to the removable path. grub-install --removable --no-nvram writes EFI/BOOT/BOOTX64.EFI (or BOOTAA64.EFI), which is what firmware boots when NVRAM has no entry for the disk. We cannot write the target machine\u0026rsquo;s NVRAM from a build host anyway, and cloud firmware is generally starting from a blank slate.\nOn aarch64 the BIOS partition is omitted and only UEFI is set up: Alpine ships grub-efi for arm64 but there is no grub-bios, because the i386-pc target is x86-only.\nThe MBR Variant Some providers\u0026rsquo; image import still rejects GPT. PARTITION_TABLE=mbr handles that:\n1 2 3 4 5 6 7 DOS/MBR # start type mount contents - sector 0 MBR + gap - grub boot.img (sector 0) and core.img (the ~1 MiB gap before p1) 1 2048 0xEF, bootable /boot same as above 2 after p1 0x83 / root No dedicated BIOS boot partition is needed, because GRUB\u0026rsquo;s i386-pc target embeds core.img in the gap between the MBR and the first partition — which is about 1 MiB, given the 2048-sector start. UEFI still works on most firmware, since the ESP is located by partition type 0xEF. That is a widely followed convention rather than something the spec promises, so MBR mode is \u0026ldquo;BIOS guaranteed, UEFI very likely\u0026rdquo;, and GPT stays the default.\nWhat It Trusts The bootstrap is a single alpine-minirootfs tarball, resolved from latest-releases.yaml for the branch:\nTLS gets the file from the mirror. Alpine publishes a .sha256 sidecar next to every release artifact. The build fails and deletes the download on a mismatch. Everything after that is apk, with Alpine\u0026rsquo;s signing keys, from the tarball\u0026rsquo;s own keyring. I use the minirootfs rather than a pinned apk.static deliberately: a hardcoded apk.static checksum drifts out of sync with the branch, and the in-image apk always matching the branch matters now that 3.23+ ships apk-tools 3.\nALPINE_BRANCH=latest-stable is the default and gets resolved once, at build time. The repositories written into the image always point at the concrete branch (v3.24), never at latest-stable, so a later apk upgrade on the running machine does not silently jump to the next stable release.\nConfiguration Copy the example and edit it; ./config.sh is picked up automatically.\n1 2 3 cp config.example.sh config.sh $EDITOR config.sh sudo ./mkalpine.sh Every variable is : \u0026quot;${VAR:=default}\u0026quot; in the script, so the environment works too, which is what I use for one-offs:\n1 sudo IMAGE_SIZE=2G ROOT_FS=xfs ./mkalpine.sh out.img Secrets default to nothing. ROOT_PASSWORD_HASH=\u0026quot;\u0026quot; locks the root account, so the failure mode of forgetting to set it is \u0026ldquo;cannot log in\u0026rdquo;, not \u0026ldquo;known root password\u0026rdquo;, and the example hash ships commented out next to the openssl passwd -6 that generates one. config.sh and the image files are in .gitignore.\nIf the build host reaches the internet through a proxy, an exported http_proxy / https_proxy / no_proxy is picked up as-is, and BUILD_HTTP_PROXY / BUILD_HTTPS_PROXY / BUILD_NO_PROXY set one for the build alone. It covers both halves of the download: the minirootfs tarball on the host, and apk plus whatever a hook fetches inside the chroot. Both cases of each name are set in the chroot\u0026rsquo;s environment, because curl reads only the lowercase http_proxy while apk and git take either — and they go into the environment rather than a profile script, so nothing about the proxy survives into the image. A proxy reachable from the build host is usually not reachable from wherever the image is deployed, and the URL frequently carries credentials.\nChoosing A Root Filesystem ROOT_FS takes btrfs, ext4 or xfs, each with its own mkfs and mount option strings:\nROOT_FS ROOT_MKFS_OPTS ROOT_MOUNT_OPTS btrfs -L alpine-root -K rw,noatime,compress=zstd:3,ssd,discard=async,space_cache=v2 ext4 -L alpine-root -m 1 -E nodiscard rw,noatime,commit=60 xfs -L alpine-root rw,noatime,logbsize=256k -K and -E nodiscard skip the discard pass at mkfs time, which is pointless on a sparse file and only produces confusing errors.\nBtrfs with zstd stays the default for the same reason as in the old post — on a small disk, transparent compression is worth a lot.\nTuning mkfs For The Storage Underneath The point of exposing ROOT_MKFS_OPTS is that a cloud disk is rarely a plain disk. config.example.sh carries these examples.\nAligning to a 16 KiB ZFS zvol (volblocksize=16k):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 # ext4: the block size cannot exceed the kernel page size (4 KiB on x86_64), # so -b 16384 produces a filesystem that will not mount. Align with # stride/stripe_width instead: stride = 16K / 4K = 4. ROOT_MKFS_OPTS=\u0026#34;-L alpine-root -m 1 -E nodiscard,stride=4,stripe_width=4\u0026#34; # or, if you really do want 16 KiB allocation clusters: ROOT_MKFS_OPTS=\u0026#34;-L alpine-root -m 1 -O bigalloc -C 16384\u0026#34; # xfs: takes the stripe unit directly. ROOT_MKFS_OPTS=\u0026#34;-L alpine-root -d su=16k,sw=1\u0026#34; # btrfs: nodesize is already 16K; set the sector size explicitly if the host # page size differs from the target\u0026#39;s. ROOT_MKFS_OPTS=\u0026#34;-L alpine-root -K -s 4096\u0026#34; Aligning to RAID6 with 6 data disks and a 128 KiB chunk:\n1 2 3 4 5 6 # ext4: stride = chunk / block = 128K / 4K = 32 # stripe_width = stride * data disks = 32 * 4 = 128 ROOT_MKFS_OPTS=\u0026#34;-L alpine-root -m 1 -E nodiscard,stride=32,stripe_width=128\u0026#34; # xfs: su = chunk, sw = number of data disks ROOT_MKFS_OPTS=\u0026#34;-L alpine-root -d su=128k,sw=4\u0026#34; Tuning Mount Options 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Favour throughput over sync latency. You lose more recent writes on an # unclean shutdown, which is fine for a rebuildable instance and not fine # for a database. ROOT_MOUNT_OPTS=\u0026#34;rw,noatime,commit=60,data=writeback\u0026#34; # ext4 ROOT_MOUNT_OPTS=\u0026#34;rw,noatime,logbsize=256k,allocsize=1m\u0026#34; # xfs ROOT_MOUNT_OPTS=\u0026#34;rw,noatime,commit=120,compress=zstd:1,ssd,discard=async,space_cache=v2\u0026#34; # Favour density on a tiny disk. zstd:6 costs CPU on write; decompression # stays cheap at any level. ROOT_MOUNT_OPTS=\u0026#34;rw,noatime,compress=zstd:6,ssd,discard=async,space_cache=v2\u0026#34; # Drop discard=async and rely on the weekly fstrim job instead, if your # provider\u0026#39;s thin pool behaves badly with continuous discards. ROOT_MOUNT_OPTS=\u0026#34;rw,noatime,compress=zstd:3,ssd,space_cache=v2\u0026#34; These options are used to mount the image during the build as well as being written to /etc/fstab, so a typo fails the build instead of the first boot. There is exactly one deliberate difference between the two, and it is the subject of Compression Needs Forcing At Build Time below.\nThe Mount Option Gotcha This one cost me a while, and it is the reason the boot test in the bundle checks mount options rather than trusting them.\nROOT_MOUNT_OPTS in /etc/fstab is not enough. The root filesystem is mounted by the initramfs, long before /etc/fstab exists, and the mount -o remount,rw / that OpenRC does afterwards cannot change every option. Most are remountable, so noatime and commit=60 looked fine. XFS, however, fixes the log buffer size at initial mount:\n1 2 3 4 5 # what I asked for ROOT_MOUNT_OPTS=\u0026#34;rw,noatime,logbsize=256k\u0026#34; # what the running machine actually had rw,noatime,inode64,logbufs=8,logbsize=32k,noquota Silently downgraded to the 32k default, with nothing in dmesg about it. Mounting the same image on the build host with the same options honoured logbsize=256k, which made it look like a kernel difference rather than what it was.\nThe fix is to put the options on the kernel command line too, so that the initial mount is the one I want:\n1 2 3 ROOTFLAGS=$(echo \u0026#34;$FSTAB_ROOT_OPTS\u0026#34; | tr \u0026#39;,\u0026#39; \u0026#39;\\n\u0026#39; | grep -vx -e rw -e ro | tr \u0026#39;\\n\u0026#39; \u0026#39;,\u0026#39; | sed \u0026#39;s/,*$//\u0026#39;) [ -n \u0026#34;$ROOTFLAGS\u0026#34; ] \u0026amp;\u0026amp; CMDLINE=\u0026#34;$CMDLINE rootflags=$ROOTFLAGS\u0026#34; rw and ro are dropped because the kernel handles those separately and GRUB already passes ro. With Btrfs you end up with two rootflags= on the cmdline, because grub-mkconfig\u0026rsquo;s 10_linux detects the subvolume and emits its own; ours is appended after GRUB\u0026rsquo;s and the initramfs takes the last one.\nNow:\n1 rw,noatime,inode64,logbufs=8,logbsize=256k,noquota Btrfs Subvolumes BTRFS_SUBVOL=\u0026quot;@\u0026quot; by default. It costs three lines at build time, and retrofitting a root subvolume later means moving every file, so it is worth doing even on a machine too small to keep many snapshots. Compression is also set with btrfs property set, so it holds regardless of mount options. BTRFS_SUBVOL=\u0026quot;\u0026quot; gives the top-level layout from the old post. GRUB does not care either way, because /boot is FAT.\nCompression Needs Forcing At Build Time compress=zstd:3 does not compress everything. Btrfs decides per file, from the beginning of the file, whether compressing is worth it — and on ELF binaries it decides wrong. In a default build, 45 MiB out of 83 MiB was stored uncompressed, including libcrypto.so.3 (3.3 MiB), every grub-* tool, busybox and ld-musl. All of those compress to roughly half. btrfs property set does not help here either: a file carrying only the inode flag still goes through the same heuristic.\ncompress-force skips the decision. The build mounts the root with it, while /etc/fstab and rootflags= keep plain compress=:\n1 2 BUILD_ROOT_OPTS=$(printf \u0026#39;%s\u0026#39; \u0026#34;$ROOT_MOUNT_OPTS\u0026#34; | sed \u0026#39;s/^compress=/compress-force=/; s/,compress=/,compress-force=/\u0026#39;) That is the one place where what the build mounts deliberately differs from what the image ships. It takes 62 MiB of data on disk down to 56 MiB, the raw image from 108 MiB to 102 MiB, and df on the booted machine from 74.5 MiB used to 67.9 MiB; BTRFS_FORCE_COMPRESS=no restores the old behaviour. The build prints what it achieved —\n1 root data 55M on disk for 86M of files (64%) — because the answer depends on what the image installs, and the failure mode is otherwise silent: the mount options, and findmnt, look exactly the same whether the files got compressed or not.\nThe heuristic is left alone for the running system on purpose. It exists to avoid burning CPU on data that will not compress, and this image contains about 10 MiB of exactly that: Alpine ships kernel modules as .ko.gz, and xfs.ko.gz, btrfs.ko.gz and friends stay uncompressed whether you force it or not. Forcing only recovers the files the heuristic misjudged.\nRunning btrfs filesystem defragment -r -czstd over the finished tree is the other way to get here, and it is the worse one. It lands at 58 MiB rather than 56 MiB, the sparse image balloons to 168 MiB before fstrim claws it back, and despite the name it does not defragment anything — compressed extents cap at 128 KiB, so the extent count went up, 1996 → 2082.\nOne caveat if you ship a compressed artifact rather than the raw image: compressing inside the image makes the outer compressor\u0026rsquo;s job harder, and the outer compressor is better at it. Forcing shrank the raw image by 5.6% and grew every compressed output — raw.zst 74 → 77 MiB, raw.xz 72 → 76 MiB. If upload size is the thing you actually care about, BTRFS_FORCE_COMPRESS=no is the right setting, and the build says as much when you ask for both.\nSizing /boot BOOT_SIZE=64M. Measured usage with one kernel is 36 MiB:\n1 2 3 4 5 6 13 M vmlinuz-virt 9.4M initramfs-virt 8.3M grub/ (i386-pc and x86_64-efi modules) 6.2M System.map-6.18.48-0-virt 156K EFI/BOOT/BOOTX64.EFI 149K config-6.18.48-0-virt That leaves headroom for one kernel and no more. Raise it to 128M if you want to keep a second kernel around — linux-lts alongside linux-virt, say — or if you want the previous kernel to survive an apk upgrade so there is something to fall back to. The build prints /boot usage at the end and warns when it lands above 80%.\nOutput Formats OUTPUT_FORMATS is a space-separated list; each entry produces one file next to the raw image.\nEntry Produced by Size Note raw the build itself 102 MiB sparse; 512 MiB apparent qcow2 qemu-img convert -c -O qcow2 81 MiB compressed qcow2 raw.zst zstd -19 -T0 77 MiB best ratio for the time; widely accepted raw.gz gzip -9 80 MiB widest provider support raw.xz xz -9 -T0 76 MiB smallest, slowest The sizes are from one build of the same default btrfs image, so they are comparable to each other rather than absolute. Note how little the four compressed formats differ — 76 to 81 MiB, under 7% between the best and the worst: the root filesystem is already zstd-compressed, so the outer compressor is mostly squeezing free space. raw.gz is a perfectly reasonable default in exchange for its compatibility.\nThat is also why BTRFS_FORCE_COMPRESS cuts the other way here. Every one of these numbers except qcow2 is larger than it would be with forcing off — raw.xz most of all, 76 MiB against 72 MiB — because data that btrfs already compressed at zstd:3 in 128 KiB blocks is data xz -9 cannot compress again. Forcing wins on raw and loses on everything else, so the build prints a reminder when you ask for both.\nDrop raw from the list to keep only the compressed artifacts. For other hypervisors, convert the raw image yourself:\n1 2 3 qemu-img convert -O vmdk out.img out.vmdk # VMware qemu-img convert -O vpc out.img out.vhd # Hyper-V / Azure qemu-img convert -O vdi out.img out.vdi # VirtualBox What Must Not Survive Cloning A disk image is a template that gets cloned N times, so anything unique baked into it stops being unique. This is the part a hand-built VM image usually gets wrong, and it is why I wanted a script in the first place — I am not going to remember all of this at 1am on a provider\u0026rsquo;s web console.\nThing Why it matters What the script does /etc/ssh/ssh_host_* Every VM built from the image shares one host key. Anyone holding the image can impersonate all of them, and clients get key-mismatch warnings after the first deploy. Delete. Regenerated on first boot. /var/lib/seedrng/seed.credential (3.17+; /var/lib/random-seed before) The saved RNG seed is restored early at boot. An identical seed on every clone means the entropy pool starts identical — including for the host keys in the row above. Delete. /etc/machine-id Used to derive DHCP DUIDs and app-level instance identity, so clones can collide on DHCP leases. Truncate to zero bytes, not delete: an empty file is the documented \u0026ldquo;generate on next boot\u0026rdquo; signal, while a missing one makes some tools fail instead. /etc/resolv.conf Would otherwise ship the build host\u0026rsquo;s nameservers. Rewritten from $DNS. /var/cache/apk/*, /var/log/*, shell history Dead weight and build-host leakage. Cleared. Filesystem UUIDs do stay identical across clones. Regenerating them on first boot means rewriting fstab and grub.cfg from a running system, which is more fragile than the problem it solves — it only bites if you attach two clones to the same host.\nThe 80-firstboot hook is the other half of this. It runs in the boot runlevel, before networking and sshd, and generates what was stripped:\n1 2 head -c 16 /dev/urandom | od -An -tx1 | tr -d \u0026#39; \\n\u0026#39; \u0026gt;/etc/machine-id ssh-keygen -A It also runs after hostname, so the host keys are commented root@alpine rather than root@(none). Cosmetic, but the alternative bothered me.\nFinally, the build stamps /etc/image-release with the Alpine version, architecture, build date, builder git commit, and the resolved filesystem and hook configuration. Six months later, working out which build a running VM came from is otherwise guesswork.\nStaying Able To Get In The single most valuable property of a cloud image is that you can get into it when networking is broken, because the provider\u0026rsquo;s serial console is then your only channel.\nconsole=ttyS0,115200 on x86_64, console=ttyAMA0,115200 on aarch64, with the getty in /etc/inittab and the tty listed in /etc/securetty. Without the securetty entry, root login on that console is refused and looks exactly like a wrong password. GRUB_TERMINAL=\u0026quot;console serial\u0026quot; plus GRUB_SERIAL_COMMAND, so the bootloader is reachable over serial too. That is what lets you fix a bad kernel cmdline or boot an older kernel remotely. No quiet. Alpine\u0026rsquo;s setup-disk defaults to KERNELOPTS=quiet; on a machine whose only debugging channel is the serial console, hiding the boot log is the wrong trade. GRUB_TIMEOUT=1 rather than 0, so there is a window to interrupt. grub.cfg is generated by grub-mkconfig rather than hand-written, which matters more than it looks: Alpine\u0026rsquo;s grub package carries triggers=\u0026quot;grub.trigger=/boot\u0026quot;, so a later apk upgrade linux-virt regenerates it and the image stays bootable with no intervention. A static config would just be silently overwritten by that same trigger. There is a hand-written fallback for the case where grub-probe cannot cope with the loop device, but I have not needed it.\nHooks One ordered list, in HOOKS. Remove a name to disable it; drop a file into hooks/ and add its name to extend. Each hook is a standalone sh script run inside the chroot with the configuration exported into its environment.\n1 HOOKS=\u0026#34;10-network 20-ssh 30-chrony 40-zram 50-logtruncate 60-sysctl 70-growroot 80-firstboot\u0026#34; Hook Does 10-network /etc/network/interfaces, DHCP or static, optional real DHCPv6 20-ssh PermitRootLogin, port, keepalives, authorized_keys 30-chrony chrony with makestep 1.0 -1 and a configurable pool 40-zram zram swap sized from RAM at boot, optionally /tmp too 50-logtruncate the hourly log cap from the old post, plus a daily apk cache clean 60-sysctl zram VM tunables, BBR and fq 70-growroot one-shot service: growpart, then grow the filesystem 80-firstboot machine-id and SSH host key generation Off by default, because they are opinionated rather than essential: 90-tools (bash, coreutils, iproute2, tcpdump, htop, tmux, vim and friends — about 120 MiB), 91-ufw, 92-sshguard, 93-podman, 94-cloud-init, and 95-dotfiles (git, zsh, lsd and a dotfiles repo, with zsh as root\u0026rsquo;s login shell).\n91-ufw, 92-sshguard and 95-dotfiles are cheap enough to add at the default IMAGE_SIZE on any filesystem. The heavier three are where the compression default starts paying for itself: 90-tools + 93-podman + 94-cloud-init together fit on btrfs at 512 MiB — 274 MiB allocated — and run the 381 MiB XFS root out of space partway through installing podman. Raise IMAGE_SIZE if you want them on ext4 or XFS.\nA few implementation notes:\n20-ssh rewrites existing lines rather than appending. sshd_config takes the first occurrence of a keyword, so appending PermitRootLogin prohibit-password to a file that already contains a commented-out default works, and appending it to one that has an active setting silently does nothing. The hook edits in place.\n50-logtruncate truncates rather than renames. Keeping one .0 copy and then truncate -s 0 on the original preserves the inode, so daemons holding the file open keep writing to it. This is the script from the old post, unchanged.\n70-growroot uses growpart, not sfdisk. Growing a GPT disk also requires relocating the backup header to the new end of the device, and growpart keeps the partition\u0026rsquo;s start sector untouched, so whatever alignment the image was built with survives. After that it is btrfs filesystem resize max /, resize2fs or xfs_growfs depending on ROOT_FS, then a stamp file and rc-update del growroot default.\n91-ufw exploits the fact that ufw only touches netfilter when enabled. All the rules are recorded inside the chroot with ufw itself, and then ENABLED=yes is set in ufw.conf as the last step. No netfilter calls happen during the build.\n95-dotfiles is the only hook that needs the network for something other than the Alpine mirror. It shallow-clones DOTFILES_REPO, runs the repo\u0026rsquo;s own bootstrap.sh, and switches root\u0026rsquo;s login shell. Two parts of that were more interesting than expected.\nChanging the login shell, first: chsh is not in a minimal Alpine — it lives in shadow, which nothing else here wants — so the hook rewrites field 7 of root\u0026rsquo;s /etc/passwd line directly. Writing the new file elsewhere and cat-ing it back keeps the original inode, mode and owner, which a mv would not:\n1 2 3 awk -F: -v OFS=: -v shell=\u0026#34;$login_shell\u0026#34; \\ \u0026#39;$1 == \u0026#34;root\u0026#34; { $7 = shell } { print }\u0026#39; /etc/passwd \u0026gt;/tmp/passwd.new cat /tmp/passwd.new \u0026gt;/etc/passwd Second, my dotfiles use zsh4humans, which installs itself — plugins, plus prebuilt fzf and gitstatusd binaries — the first time an interactive zsh starts. Doing that during the build instead buys two things: the shell works on a machine with no internet, and a download failure fails the build rather than the first login.\nIt costs about 67 MiB of files — 26 MiB of packages, 5 MiB of checkout, 28 MiB of z4h cache — which is 21 MiB of actual disk on the compressed btrfs default. 99-selftest checks it the way that matters: it starts an interactive zsh on the booted machine and confirms the repo\u0026rsquo;s .zshrc really was sourced.\nBeing the only hook that talks to GitHub also makes it the one most likely to fail a build. Set http_proxy and https_proxy when the path to GitHub is reliably bad rather than occasionally bad.\nWeekly fstrim is installed regardless of ROOT_FS. Although a btrfs root already trims itself through discard=async, the job is not all about the root filesystem. A data disk attached to the instance later is quite likely to be ext4 or XFS, and nothing else in the image would ever trim it. For those, periodic trim is the currently recommended approach over -o discard.\nThe job also cannot be a one-liner, because a minimal Alpine image does not have util-linux:\n1 2 3 4 5 6 fstrim -a 2\u0026gt;/dev/null \u0026amp;\u0026amp; exit 0 awk \u0026#39;$1 ~ /^\\/dev\\// \u0026amp;\u0026amp; !seen[$2]++ { print $2 }\u0026#39; /proc/mounts | while IFS= read -r mp; do fstrim \u0026#34;$mp\u0026#34; 2\u0026gt;/dev/null done fstrim -a walks every mounted filesystem and de-duplicates devices, but that is util-linux\u0026rsquo;s fstrim. Busybox\u0026rsquo;s applet takes exactly one mount point and has no -a at all, so on the image as built the -a form fails and the loop does the work. My original fstrim -a || fstrim / looked like it handled that and did not: the fallback trimmed only the root, which is precisely the filesystem that needed it least.\nWriting Your Own Drop a script in hooks/, add its name to HOOKS. The whole hooks/ directory is copied into the chroot at /tmp/mkalpine, so a hook that needs to install a longer file can keep it in hooks/files/ and copy it from /tmp/mkalpine/files/ rather than embedding it in a heredoc:\n1 2 3 4 5 #!/bin/sh set -eu apk add --quiet --no-progress my-thing install -m 0644 /tmp/mkalpine/files/my-thing.conf /etc/my-thing.conf rc-update add my-thing default The number prefixes are a label, not a mechanism. This is not run-parts: nothing sorts the directory, and the run order is simply the order of the HOOKS list, so HOOKS=\u0026quot;10-network my-thing 80-firstboot\u0026quot; does exactly what it looks like. Your hook can be called anything, go anywhere in the list, and it does not matter that the shipped extras have crowded the 90s — 99-selftest runs last because it is last in the list, not because of its number.\nTwo things about the environment a hook runs in. Everything in config.sh is exported into it, but nothing else is: the chroot starts from env -i, so a hook sees PATH, HOME, TERM, the config, and a proxy if one is configured. And it has no controlling terminal and /dev/null on stdin, for the reasons in 95-dotfiles above, so a hook cannot stop a build to ask a question — a program that tries gets EOF and fails, which is a build that ends with an error rather than one that waits all night.\nVerifying The Image An image that builds is not an image that boots, and an image that boots is not an image that is configured the way you asked. So there are two scripts.\nDoes It Boot 1 2 ./testboot.sh -m uefi alpine.img ./testboot.sh -m bios alpine.img This boots the image headless under QEMU with -snapshot, so the image is never modified, captures the serial console to a file, and waits for a login: prompt. Reaching the login prompt exercises the whole chain in one go: firmware → GRUB → kernel → initramfs → root mount → OpenRC → getty. On failure it prints the last 40 lines of the console, which is usually enough to see where it stopped. -w drops the -snapshot and lets the guest write to the image, for when you want to inspect what a first-boot service did to the disk.\nIt finds OVMF and AAVMF across the various paths different distributions use, and uses KVM when /dev/kvm is writable. With KVM, BIOS reaches the login prompt in about 16 seconds and UEFI in about 26 — OVMF initialization is the difference.\nIs It Configured Correctly Add the 99-selftest hook and pass -d:\n1 2 HOOKS=\u0026#34;$HOOKS 99-selftest\u0026#34; sudo ./mkalpine.sh -f alpine.img ./testboot.sh -d alpine.img The hook installs a service that runs last in the default runlevel and checks the running system against the configuration it was built from. The expectations are baked in at build time, which is the part that makes it useful: a hook that silently did nothing shows up as a failure rather than as a plausible-looking dump.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 ===== SELFTEST BEGIN ===== -- identity info machine-id=2767df9cefbfd2d2af2d0b8acdfb4aa5 info hostkey=SHA256:U242JZNiEf+Qo3swIK+MoW4m2bdNb2eCMG5W6VJAwPM (ED25519) ok hostname is alpine ok machine-id is 32 hex digits ok timezone is UTC -- filesystems info root options=rw,noatime,inode64,logbufs=8,logbsize=256k,noquota ok root is xfs ok /boot is vfat ok root mounted with logbsize=256k -- btrfs subvolume (not configured) ok fstab references root by UUID -- boot info kernel=6.18.48-0-virt info cmdline=BOOT_IMAGE=/vmlinuz-virt root=UUID=e95aee24-... ro modules=sd-mod,usb-storage,xfs rootfstype=xfs rootflags=noatime,logbsize=256k console=tty0 console=ttyS0,115200 ok cmdline is not quiet ok ttyS0 in securetty -- hooks ok 20-ssh: PermitRootLogin is prohibit-password ok 20-ssh: listening on 22 ok 40-zram: swap is ~100% of RAM ok 60-sysctl: congestion control is bbr ok 70-growroot: removed itself from the default runlevel ok 70-growroot: root fs fills the partition ok 80-firstboot: host key is not the build host\u0026#39;s -- 91-ufw (disabled) -- hygiene ok apk cache is empty ok no build scratch left behind ok repositories pinned to v3.24 ok repositories do not track latest-stable -- sizing Filesystem Size Used Available Use% Mounted on /dev/vda3 381.0M 161.2M 219.8M 42% / /dev/vda2 63.0M 36.4M 26.6M 58% /boot SELFTEST RESULT: 59 passed, 0 failed ===== SELFTEST END ===== (abridged; the real report is one line per check)\ntestboot.sh -d exits non-zero if any check failed, so it works in a loop.\nThe checks live in hooks/files/selftest.initd as an ordinary shell script rather than a heredoc, and check takes a command with its arguments instead of a string to eval:\n1 2 3 4 check \u0026#34;root is $EXPECT_ROOT_FS\u0026#34; fstype_is / \u0026#34;$EXPECT_ROOT_FS\u0026#34; check \u0026#34;machine-id is populated\u0026#34; test -s /etc/machine-id check \u0026#34;cmdline is not quiet\u0026#34; not grep -qw quiet /proc/cmdline check \u0026#34;60-sysctl: vm.swappiness is 180\u0026#34; output_is 180 sysctl -n vm.swappiness Anything needing a pipeline gets a named predicate instead. The first version of this used eval on quoted strings and was unreadable — check \u0026quot;root is btrfs\u0026quot; \u0026quot;awk '\\$2 == \\\u0026quot;/\\\u0026quot; {print \\$3}' /proc/mounts | grep -qx btrfs\u0026quot; — which is a good sign that the abstraction was wrong.\nEverything At Once 1 sudo ./testmatrix.sh -o /var/tmp/mx This builds btrfs/ext4/xfs × gpt/mbr, boot-tests all six under both BIOS and UEFI with the selftest enabled, then checks the things that only show up at runtime:\nGrowth. Copy an image, truncate -s 4G, and boot it — with testboot.sh -w, so the guest\u0026rsquo;s writes actually land in the file instead of in QEMU\u0026rsquo;s -snapshot overlay. Then confirm the root partition grew (911,360 → 8,253,407 sectors) and that it still starts on its original sector, because a growpart that moved the start would silently discard the build-time alignment. Reading the table back from the un-booted copy would have made this test tautological, which is what it was until I looked closely. Output formats. Build with all five, confirm each file exists, and boot the qcow2 directly rather than just checking that qemu-img produced something. Identity. Boot the same image twice and confirm the two boots produce different SSH host key fingerprints and machine-ids. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 == build and boot matrix == ok build btrfs/gpt ok boot btrfs/gpt/uefi ok boot btrfs/gpt/bios ... == growroot on a larger disk == ok grow 4G ok grow root partition 1959936 -\u0026gt; 8253407 sectors, still at 135168 == output formats == ok format formats.img.zst (74M) ok boot qcow2 == identity is per-boot, not per-image == ok hygiene two boots produced different SSH host keys ok hygiene two boots produced different machine-ids == summary == 29 passed, 0 failed Four real bugs in this post were found this way. The XFS logbsize downgrade above was one, and the tautological growth check was another. The third was in the selftest itself: I had written\n1 check \u0026#34;apk cache is empty\u0026#34; not ls -A /var/cache/apk which reads correctly and means the opposite, because ls -A exits 0 on an empty directory. A test suite that catches bugs in its own checks is doing its job.\nThe last one showed up when I dropped the default IMAGE_SIZE from 1 GiB to 512 MiB. All three filesystems still built and booted, but ext4 and XFS started failing 70-growroot: root fs fills the partition, which had been written as \u0026ldquo;df size is at least 90% of the partition\u0026rdquo;. Nothing was wrong with the images: df excludes reserved blocks and fixed metadata, and on a 445 MiB root partition that fixed cost is 44 MiB for ext4 and 64 MiB for XFS — 90.0% and 85.6%. The same filesystems on a 957 MiB partition reported 96.5% and 93.3% and passed. A percentage was simply the wrong shape for the check, because the overhead it has to tolerate barely moves with the disk size while the threshold does. The failure it exists to catch is not marginal either — a root that never grew is a 445 MiB filesystem in a 4 GiB partition — so the tolerance is now the larger of 96 MiB and 10%.\nUploading Most providers want a compressed raw image or a qcow2:\n1 sudo OUTPUT_FORMATS=\u0026#34;raw.zst\u0026#34; ./mkalpine.sh -f alpine.img If your provider has no image import at all — which is the common case on the cheap tiers — the restore-over-dd trick from the old post still applies: boot their stock OS, dd the image onto the disk from a rescue environment or over SSH, and reboot. 70-growroot then handles the fact that their disk is larger than the image.\nThat flow needs a console or a rescue system at one point or another. When the provider offers neither — no VNC, no serial, no rescue mode, just the stock OS and SSH, which is what the smallest NAT\u0026rsquo;d tiers look like — use the no-console variant in the old post instead: reinstall dd mode arranges a RAM-resident Alpine by rewriting the stock bootloader, then writes the image over the whole disk and reboots, all over SSH. It accepts the raw.gz / raw.zst / raw.xz artifacts from OUTPUT_FORMATS as-is, and it does not modify a Linux image — which, on a headless box, promotes 10-network and 20-ssh from conveniences to the difference between a machine that comes back and a reinstall ticket.\nWhat I Kept From The Old Post Everything about why. The old post explains the 64 MiB /boot, the compressed Btrfs root, the absence of disk swap, and the individual tweaks in far more detail than a config file comment can, and it is still the reference for doing this by hand on a machine you have already booted. This post is the automation of it.\n","date":"2026-09-05T22:00:00+08:00","permalink":"https://charlie0129.github.io/blog/p/alpine-image-builder/","title":"Building Alpine Linux Disk Images Without a VM"},{"content":"I like audio analyzers. It answers questions that my ears alone cannot answer quickly. Where is that resonance? Is the low end actually mono? How loud is this over the whole track? Is a limiter catching an occasional peak, or working all the time?\nWhat I do not like is an analyzer whose interface feels slower than the display it runs on.\nOn my M1 Max, Excite Audio VISION 4X appeared to top out at roughly 30 FPS, with inconsistent frame timing, while consuming about one CPU core. iZotope Insight 2 looked smoother, but in my experience it was comparatively resource-heavy and expensive. These were observations from my own setup, not controlled benchmarks that apply to every machine, host, and plugin version. Still, they were enough to make me wonder: how difficult would it be to build the analyzer I wanted?\nThat became Audio Insight, an open-source AUv2 and VST3 analyzer for macOS. Its first goal is deliberately narrow: show useful measurements, leave the audio unchanged, keep real-time callback work bounded, and make the interface feel native on a high-refresh-rate display.\nThe project is still early, but it already has a Spectrum, Spectrogram, Peak/RMS meter, stereo vectorscope and correlation meter, and BS.1770 loudness measurements. Four grid-snapped splitters resize the dashboard tiles, the analysis parameters are adjustable, and a built-in metrics panel makes the renderer\u0026rsquo;s behavior visible instead of leaving performance to intuition.\nThis post is about how it works, but mostly about the unexpectedly interesting work required to make a meter move smoothly.\nWhat an audio plugin actually does People who use plugins often picture them as little applications inside a DAW. That is a useful mental model for the interface, but not for the audio path.\nAn AU or VST3 plugin is code loaded by a host (or, in some hosts, a separate hosting service). The host repeatedly gives the plugin a small block of samples by calling its processing function. At 48 kHz with 512-sample blocks, a new block arrives about every 10.7 milliseconds. The plugin has to finish before the hardware needs the result. Missing that deadline can produce a click or dropout.\nAudio Insight is a transparent effect: it observes supported mono or stereo audio and leaves the samples unchanged. Even so, its callback has to follow the same real-time rules as a compressor or synthesizer. It cannot allocate memory, take a lock, wait for another thread, write a log, open a file, call the UI, or ask the GPU to draw something. Any of those operations can take an unpredictable amount of time.\nThe resulting design looks like this:\n1 2 3 4 5 6 7 8 9 10 11 host audio callback ↓ bounded, non-blocking capture ↓ per-instance analysis coordinator ↓ shared two-worker analysis pool ↓ immutable measurement snapshots ↓ display-linked Metal renderer The callback only captures bounded chunks into preallocated storage and updates the few measurements that must inspect every sample. A per-instance coordinator coalesces work, and all instances loaded in the same plugin module share two analysis workers. There is at most one running or queued job per instance, so opening many plugin windows does not create a thread for every visualization.\nThe workers publish immutable snapshots. The UI reads the newest complete snapshot whenever it draws; it never waits for analysis to finish. This separation matters. Analysis targets 60 slices per second, but the latest-wins scheduler can skip stale work rather than build a backlog. Meanwhile, a ProMotion display can render at around 120 Hz. The renderer can advance display motion between discrete analysis updates without running twice as many FFTs.\nAll five visualizations share one Metal canvas, drawable, command buffer, and render pass. JUCE supplies the plugin shell and cross-format plumbing, while the visual layer is native Metal. Coordinates and layout use logical points, and drawable and text resources follow the current backing scale. The implementation is therefore designed to support both regular-density and Retina displays, including live backing-scale changes.\nBoth paths are highly optimized. The shared transform uses juce::dsp::FFT, which selects Apple\u0026rsquo;s Accelerate/vDSP implementation on macOS, and Spectrum and Spectrogram reuse each calibrated result. On the GPU side, the Spectrogram stores calibrated dB in a circular 16-bit-float (R16Float) texture: scrolling remaps texture coordinates, while shader controls recolor retained history without another FFT or a whole-texture copy. High display cadence therefore does not multiply the default 60 Hz FFT workload.\nWhen the editor is closed, there is nothing to display, so capture, analysis, history, display-link callbacks, and Metal submissions stop. Audio still passes through normally. Reopening the editor begins fresh rather than silently spending host resources on invisible history.\nTurning samples into pictures The analyzers share infrastructure, but each one answers a different question. Here is the calculation path in a little more detail.\nSpectrum: what frequencies exist now? The Spectrum takes a short window of recent audio and uses a fast Fourier transform (FFT) to divide it into frequency bins. By default the transform size is \\(N=8192\\) samples, or about 171 milliseconds at a sample rate \\(F_s=48\\) kHz. This does not delay the audio by 171 ms; the plugin passes audio through immediately. It means the displayed estimate describes roughly that much recent history.\nThe FFT bin centers are separated by \\(\\Delta f=F_s/N\\). With the defaults, that is approximately 5.86 Hz. This number is useful, but it is not the same as saying two tones 5.86 Hz apart can always be resolved: the selected window also determines how widely a tone spreads into nearby bins.\nBefore the FFT, samples are multiplied by a periodic five-term flat-top window \\(w[n]\\). Cutting an arbitrary piece from a continuous waveform creates artificial edges, which spread energy across the spectrum. A window tapers the data to control that leakage. A flat-top window trades some ability to separate nearby tones for better amplitude accuracy, which is a useful default for a measurement tool.\nFor channel \\(c\\), the transform is:\n\\[ \\begin{aligned} X_c[k]\u0026=\\sum_{n=0}^{N-1}x_c[n]\\,w[n]e^{-j2\\pi kn/N},\\\\ f_k\u0026=\\frac{kF_s}{N}. \\end{aligned} \\]Audio Insight corrects the window\u0026rsquo;s coherent gain—the amplitude scaling introduced by multiplying by \\(w[n]\\)—with \\(W=\\sum_n w[n]\\). Real-valued audio has mirrored positive- and negative-frequency FFT bins, but the graph needs only the nonnegative half. In this one-sided view, the DC bin at 0 Hz and the Nyquist bin at \\(F_s/2\\) use \\(1/W\\); every bin between them represents both mirrored sides and uses \\(2/W\\). The calibrated stereo power and level are therefore:\n\\[ \\begin{aligned} a_k\u0026= \\begin{cases} 1/W, \u0026 k=0\\ \\text{or}\\ k=N/2,\\\\ 2/W, \u0026 \\text{otherwise}, \\end{cases}\\\\[3pt] P[k]\u0026=\\max_c\\left(a_k|X_c[k]|\\right)^2,\\\\ D[k]\u0026=10\\log_{10}P[k]. \\end{aligned} \\]For mono, the maximum contains only one channel. For stereo, taking the larger channel magnitude avoids first mixing the waveforms to mono, where out-of-phase content could cancel. The calibration makes a bin-centered full-scale sine read 0 dB internally; powers at or below \\(10^{-18}\\) are displayed at the \\(-180\\) dB analysis floor.\nAttack and Release then smooth each bin in linear power, not in dB. Given the elapsed time \\(\\Delta t\\) and the selected time constant \\(\\tau_d\\):\n\\[ \\begin{aligned} \\alpha_d\u0026= \\begin{cases} 0, \u0026 d\\text{ is Off},\\\\ e^{-\\Delta t/\\tau_d}, \u0026 d\\text{ is enabled}, \\end{cases}\\\\[3pt] \\bar P_t[k]\u0026=\\alpha_d\\bar P_{t-1}[k]+(1-\\alpha_d)P_t[k]. \\end{aligned} \\]The direction \\(d\\) is Attack when \\(P_t[k]\\geq\\bar P_{t-1}[k]\\), otherwise Release. An Off direction follows the current FFT immediately. The default Attack is Off, allowing a short burst to appear at once, while the default 250 ms Release lets the trace fall more slowly. Peak hold, when enabled, operates on unsmoothed power instead of \\(\\bar P\\).\nTransforms target a slice rate \\(R_s\\) using a hop of \\(H=\\max(1,\\operatorname{round}(F_s/R_s))\\) new samples. At 48 kHz and 60 slices per second, \\(H=800\\), so adjacent 8,192-sample windows overlap by about 90.2%. The first result still waits for one complete window, and the latest-wins scheduler may skip stale transforms under load instead of building a backlog.\nSpectrum and Spectrogram use the same continuously adjustable frequency scale. For a frequency \\(f\\) between \\(f_0\\) and \\(f_1\\), the scale control \\(s\\) blends normalized linear and logarithmic coordinates:\n\\[ \\begin{aligned} u_{\\mathrm{lin}}(f)\u0026=\\frac{f-f_0}{f_1-f_0},\\\\ u_{\\log}(f)\u0026=\\frac{\\ln(f/f_0)}{\\ln(f_1/f_0)},\\\\ u(f,s)\u0026=(1-s)u_{\\mathrm{lin}}(f)+s\\,u_{\\log}(f). \\end{aligned} \\]Here \\(f_0=20\\) Hz and \\(f_1=\\min(20\\text{ kHz},F_s/2)\\). The default is \\(s=0.8\\). At \\(s=0\\), equal distances represent equal numbers of hertz. At \\(s=1\\), equal ratios such as 100→200 Hz and 1→2 kHz occupy equal distances. Values in between preserve more low-frequency detail without compressing the entire treble into a tiny area. Spectrum uses \\(x=u\\), while Spectrogram uses \\(y=1-u\\) so high frequencies appear at the top. Axis labels are chosen dynamically: important anchors win first, then extra candidates fill only the space that remains.\nSpectrogram: how did the spectrum change? A Spectrum is one slice through time. A Spectrogram keeps those slices and scrolls them sideways, using color for level. Transients become vertical marks, steady tones become horizontal lines, and harmonics become stacks of related lines.\nEach Spectrogram column starts from the same raw power \\(P[k]\\) as Spectrum, before Spectrum\u0026rsquo;s Attack/Release averaging. Let \\(\\mathcal K\\) contain only usable bin centers from 20 Hz through \\(f_1\\), and let \\(R_f=\\min(1024,|\\mathcal K|)\\) be the texture\u0026rsquo;s frequency-row count. For a usable bin at \\(f_k=kF_s/N\\), define \\(q_k=u(f_k,s)\\). Its row is:\n\\[ r(k)=\\min\\left(R_f-1,\\left\\lfloor R_f q_k\\right\\rfloor\\right). \\]For a row containing one or more bin centers, \\(P_r\\) is the greatest \\(P[k]\\) assigned to that row. Taking the maximum, rather than the average, helps a narrow tonal trace survive when several FFT bins land in one display row. If a row contains no bin center—common at low frequencies with a small FFT—the mapper inverse-maps the row center and linearly interpolates the two surrounding bins in power. It is honest interpolation between available samples, not a claim of extra frequency resolution.\nPower at or below \\(10^{-18}\\) becomes \\(-180\\) dB; otherwise the mapper stores \\(D_r=10\\log_{10}P_r\\). These values go into a circular Metal texture with one 16-bit floating-point level per cell (R16Float). The texture stores calibrated dB rather than finished colors. In the shader, let \\(F\\) be the selected floor, \\(C\\) the ceiling, and \\(\\eta\\) the Color response:\n\\[ \\begin{aligned} v\u0026=\\operatorname{clamp}\\left( \\frac{D_r-F}{C-F},0,1\\right),\\\\ \\gamma\u0026=2^\\eta,\\\\ c_{\\mathrm{palette}}\u0026=v^\\gamma. \\end{aligned} \\]The value \\(c_{\\mathrm{palette}}\\) selects a point in the chosen palette. Response 0 is linear in dB; negative values reveal quieter detail, while positive values suppress low energy and emphasize stronger traces. Because this work happens in the shader, changing palette, range, or response recolors existing history without rerunning the FFT.\nFor a history duration \\(T\\) and requested slice rate \\(R_s\\), the texture uses \\(\\min(8192,\\lceil TR_s\\rceil)\\) columns. The default ten seconds at 60 slices per second therefore needs 600 columns. A write index wraps around the texture, and the renderer changes texture coordinates instead of copying the whole image to scroll it. Missing timestamp intervals become black columns rather than stretching old information across time.\nPeak and RMS: how strong is the signal? Peak and RMS intentionally describe different things.\nSample peak examines every sample. Its live value has instantaneous attack and a 20 dB/s release:\n\\[ \\begin{aligned} \\lambda_p\u0026=10^{-20/(20F_s)},\\\\ p[n]\u0026=\\max\\left(|x[n]|,\\lambda_p p[n-1]\\right). \\end{aligned} \\]In other words, a new larger sample wins immediately; otherwise the old indication decays by the amount corresponding to one sample period. A separate hold marker keeps a new maximum for two seconds, then falls at the same 20 dB/s rate. The OVER indicator latches when \\(|x[n]|\\geq1\\), although the label deliberately does not claim that floating-point audio at 0 dBFS proves waveform clipping.\nRMS estimates sustained signal power. With the 300 ms time constant \\(\\tau=0.300\\) s, Audio Insight updates an exponential mean square for every sample:\n\\[ \\begin{aligned} \\alpha\u0026=e^{-1/(F_s\\tau)},\\\\ q[n]\u0026=\\alpha q[n-1]+(1-\\alpha)x[n]^2,\\\\ \\operatorname{RMS}[n]\u0026=\\sqrt{q[n]},\\\\ D_{\\mathrm{RMS}}[n]\u0026=20\\log_{10}\\operatorname{RMS}[n]. \\end{aligned} \\]This is an exponential response, not a rectangular box containing exactly the latest 300 ms. It also has no AES17 \\(+3.01\\) dB calibration offset, so a full-scale sine reads approximately \\(-3.01\\) dBFS RMS. Peak reveals brief extremes; RMS behaves more like a view of sustained energy. The peak remains a sample peak, not an oversampled true-peak/dBTP measurement, so it does not predict a possibly larger value between stored samples.\nThese ballistics run on the bounded real-time capture path and inspect every sample. Their meaning therefore does not change if an analysis worker is briefly late.\nStereo: how are left and right related? The vectorscope turns each stereo sample pair into a point:\n\\[ x_{\\mathrm{scope}}=\\frac{R-L}{2}, \\qquad y_{\\mathrm{scope}}=\\frac{L+R}{2}. \\]Audio shared equally by both channels has \\(x=0\\) and lies on the vertical center axis. Opposite-phase audio has \\(y=0\\) and spreads horizontally. The coordinates remain tied to full scale rather than being normalized independently on every frame, so a quiet signal is not made to look artificially loud.\nThe field keeps the latest 250 ms but bounds its GPU data. For \\(W_f=\\lceil0.25F_s\\rceil\\) captured frames, the worker selects one pair every:\n\\[ d=\\left\\lceil\\frac{W_f}{4096}\\right\\rceil \\]frames. At 48 kHz, \\(W_f=12000\\), \\(d=3\\), and the cloud contains about 4,000 uniformly spaced points. Their opacity fades with age. This decimation changes only the picture; it does not change the correlation measurement.\nThe adjacent correlation value uses every sample, with 300 ms exponentially weighted averages:\n\\[ \\begin{aligned} \\alpha\u0026=e^{-1/(F_s\\cdot0.300)},\\\\ E_n[z]\u0026=\\alpha E_{n-1}[z]+(1-\\alpha)z[n]. \\end{aligned} \\]\\[ \\rho[n]=\\frac{E_n[LR]}{\\sqrt{E_n[L^2]E_n[R^2]}}. \\]The three running values \\(E[L^2]\\), \\(E[R^2]\\), and \\(E[LR]\\) advance in source-sample order on the real-time side, and the implementation clamps the final ratio to \\([-1,1]\\) against numerical error. A value near \\(+1\\) means the channels are strongly alike, \\(0\\) means little linear relationship, and a negative value warns that mono playback may cancel important content. If either averaged channel power is below \\(10^{-9}\\), equivalent to \\(-90\\) dBFS RMS, correlation is reported as unavailable rather than dividing by a nearly zero value. A genuinely mono input is labeled MONO and likewise does not receive a synthetic \\(+1\\) correlation.\nLoudness: how loud does it feel over time? Raw peak level is not perceived loudness. Audio Insight implements BS.1770-5 K-weighting with the Momentary, Short-term, and Integrated semantics commonly used with EBU R128. K-weighting is a pair of filters: a high-frequency shelf models the head\u0026rsquo;s acoustic effect, and a high-pass stage reduces the contribution of very low frequencies. The code derives their coefficients for the current sample rate.\nIf \\(y_c[n]\\) is the K-weighted output of channel \\(c\\), Audio Insight forms the per-sample energy and a window mean:\n\\[ \\begin{aligned} e[n]\u0026=\\sum_c y_c[n]^2,\\\\ z_W\u0026=\\frac{1}{N_W}\\sum_{n\\in W}e[n]. \\end{aligned} \\]For the supported mono and stereo layouts, every actual channel has unit weight. Mono therefore contributes once; it is never duplicated into synthetic left and right channels. Surround layouts and their channel weights are outside the current scope. Window energy becomes LUFS (Loudness Units relative to Full Scale) using the BS.1770 offset:\n\\[ L_W=-0.691+10\\log_{10}z_W. \\] Momentary loudness covers 400 ms. Short-term loudness covers 3 seconds. Integrated loudness uses 400 ms blocks completed every 100 ms—75% overlap—from the latest Reset within the current uninterrupted, visible analysis interval. Momentary and Short-term are simple ungated window measurements. Integrated loudness applies gates: thresholds that exclude blocks from the long-term average. For each 400 ms block \\(i\\), let its mean-square energy be \\(z_i\\) and its loudness be \\(L_i=-0.691+10\\log_{10}z_i\\). The absolute-passing set is:\n\\[ \\begin{aligned} \\mathcal A\u0026=\\{i\\mid L_i\u003e-70\\ \\mathrm{LUFS}\\},\\\\ \\mu_{\\mathcal A}\u0026=\\frac{1}{|\\mathcal A|}\\sum_{i\\in\\mathcal A}z_i. \\end{aligned} \\]At each Integrated update, one non-iterative relative threshold is calculated 10 LU below the preliminary absolute-gated mean of the history accumulated so far:\n\\[ \\begin{aligned} \\Gamma_{\\mathrm{rel}}\u0026=-0.691+10\\log_{10}\\mu_{\\mathcal A}-10,\\\\ \\mathcal R\u0026=\\{i\\in\\mathcal A\\mid L_i\u003e\\Gamma_{\\mathrm{rel}}\\}. \\end{aligned} \\]Finally:\n\\[ \\begin{aligned} \\bar z_{\\mathcal R}\u0026=\\frac{1}{|\\mathcal R|}\\sum_{i\\in\\mathcal R}z_i,\\\\ L_I\u0026=-0.691+10\\log_{10}\\bar z_{\\mathcal R}. \\end{aligned} \\]Both comparisons are strict \\(\u003e\\), and the relative gate is not iterated repeatedly. This two-stage gate prevents silence and very quiet passages from dragging the program average down indefinitely. The tile\u0026rsquo;s Reset command restarts Integrated loudness while ready Momentary/Short-term values and K-weighting continuity remain intact. Editor reactivation, an audio discontinuity, or a format change resets the complete loudness analyzer.\nThe empty cases are explicit too. If \\(\\mathcal A\\) contains no blocks, the preliminary mean and relative gate remain unavailable. If \\(\\mathcal R\\) is empty, Integrated loudness remains \\(-\\infty\\). The implementation never divides by an empty set.\nThe implementation does not claim complete EBU Mode compliance: it does not yet include LRA or true peak, for example. The label describes its M/S/I measurement semantics, not a certification.\nThere is an interesting performance problem hiding in Integrated loudness. Within one uninterrupted visible measurement, the exact answer can cover 24 hours: up to 864,000 blocks. Rescanning every qualifying block every 100 ms would make the cost grow throughout the measurement.\nThe implementation uses a preallocated sorted index called a B+ tree. It contains finite block energies above the absolute gate and keeps aggregate counts and sums in its branches; all completed blocks still count toward the 24-hour limit. A new relative-gate boundary can be answered by finding one boundary leaf and combining a bounded number of branch totals. Capacity for the worst case occupies about 7.25 MiB on arm64, and the structure never allocates while processing.\nSmooth is a timing property, not an FPS number The renderer uses CAMetalDisplayLink, which supplies a drawable in step with a display. While visible, Audio Insight requests the active display\u0026rsquo;s exact reported maximum refresh rate, with a 60 Hz fallback. That request is best effort—Core Animation and the compositor still control actual presentation—so measured presentation timestamps are the truth.\nThis distinction became important repeatedly. A counter can say 120 callbacks per second while the screen still changes only 60 times. An average can say 120 FPS while an occasional doubled interval makes scrolling visibly hitch. Smoothness is about the complete chain from callback to presentation and about the distribution of frame intervals, not just one large number.\nThe plugin that crashed its host The first AU build appeared for a moment in SoundSource and then disappeared. The host reported only that its Audio Unit hosting service had crashed.\nThe detailed log led to an assertion in timed drawable presentation. A normal Metal application may call an API such as timed present, but a drawable delivered by CAMetalDisplayLink has different presentation ownership. Combining the two caused the hosting process to assert. The correct sequence is to commit the command buffer and call plain present() on that drawable, while using the display link\u0026rsquo;s target timestamp only for telemetry and scheduling.\nThis is one reason plugin development needs testing in real hosts. SoundSource exposed an API misuse that a successful build or unit test had not.\nWhy 120 display callbacks produced 60 frames After the crash was fixed, the display link was firing close to 120 times per second, but only about 60 frames were submitted. The built-in metrics captured the pattern:\nCounter Before the fix Display-link callbacks 6,453 Metal submissions 3,230 GPU-backpressure drops 3,223 Sampled display-link callback rate ~111/s Sampled Metal submission rate ~59.6/s The cumulative counters cover the full telemetry epoch; the two rates are a sample from its final roughly 0.25 seconds. Almost exactly every other display-link callback was being rejected by the in-flight buffer pool.\nThe surprising part was that the GPU was not necessarily too slow. Reusable vertex buffers were retained until the drawable was actually presented. The compositor may hold a drawable for several refresh periods even after GPU execution has completed, so all reusable buffers became occupied and the next callback had nowhere to write.\nThe fix was to separate two lifetimes. GPU command completion now releases the reusable buffers immediately. A small, independent object survives only to correlate the later presentation timestamp. The renderer no longer holds large working resources hostage to compositor timing.\nIn a later point-in-time M1 Max capture from another development build, with the Metrics panel visible, the drawable was 2,400×1,496 pixels at 2× backing scale. The run recorded 1,188 display-link callbacks, 1,188 submissions, and zero GPU-backpressure drops. Across the most recent 240 presented intervals, the average was 8.438 ms, or 118.52 Hz; 237 intervals were the normal 8.333 ms and three doubled to 16.667 ms. Telemetry also counted 11 skipped presentations over the run. That capture used the then-selected 16,384-point FFT; today\u0026rsquo;s default is 8,192.\nThe capture demonstrates that the buffer-lifetime bottleneck and its GPU-backpressure drops were gone. It is evidence of approximately display-rate presentation in that run, not a perfect-pacing claim or a controlled comparison with another plugin.\nMaking 60 Hz data scroll on a 120 Hz display The Spectrogram exposed a second kind of stutter. New analysis columns arrive at 60 Hz. If the image moves forward only when a complete column arrives, it necessarily steps every other frame on a 120 Hz display.\nThe solution was not to double the FFT workload. The renderer advances a fractional scroll head from the target presentation clock while keeping the actual dB cells discrete. A one-slice cushion absorbs ordinary analysis scheduling jitter. If a texture upload is briefly busy, the renderer postpones that upload while continuing to draw the rest of the dashboard.\nThe result is much smoother motion from the same 60-slices-per-second target. This also explains why raising thread priority would have been the wrong first response: the main issue was the relationship between two clocks, not a shortage of real-time privileges.\nThe random resets that were not random During longer sessions, all graphs would occasionally reset. The recovery was intentional—when audio history has a real gap, temporal analyzers must not pretend the samples on either side were adjacent—but the handoff overflowed far too easily even while host audio was continuous. Sequence tracking then correctly detected the resulting loss.\nThe original capture queue had 16 logical slots and consumed one for each host callback. Its time capacity therefore depended on the host\u0026rsquo;s block size. A metrics capture reached all 16 ready slots, discarded 20 queued chunks to make room for newer audio, and recorded three consumer discontinuities followed by three Loudness resets.\nThe redesigned queue packs audio across callback boundaries into 128 slots of 256 frames, retaining 32,768 frames regardless of host callback size. That is about 683 ms at 48 kHz, 341 ms at 96 kHz, or 171 ms at 192 kHz. The capacity and overflow behavior are covered by implementation tests; longer post-redesign host runs remain part of validation. A sufficiently long stall can still overflow it. When that happens, latest data wins and temporal analyzers reset, because joining unrelated pieces of audio would produce convincing but false measurements.\nBuilding observability into the plugin Apple\u0026rsquo;s Metal HUD is useful for applications that enable it before creating their first Metal device. A plugin usually arrives after its host has already done that, so it cannot reliably switch the HUD on from a settings button. I replaced that idea with a built-in performance panel available in Release builds.\nIt reports exact frame pacing over the latest 240 presentation intervals, derived from 241 timestamps; CPU, submit, GPU, and compositor latency composition; display-link scheduling; audio-callback histograms; queue occupancy and discontinuities; analyzer freshness; and raw copyable metrics for offline inspection. The stacked latency bar covers the pipeline from a display-link callback to presentation. It is not a breakdown of an 8.33 ms frame budget: several frames can overlap in flight, so its total can exceed one refresh interval without reducing presentation cadence.\nGraphs move at vblank, headline numbers refresh at no more than 10 Hz, and the full text table refreshes at 4 Hz. That keeps the visual feedback immediate without rebuilding lots of text 120 times per second. Instrumentation turned several vague reports—“it looks a bit laggy,” “it seems to reset”—into specific, actionable failures.\nAn agent-assisted, human-tested development loop I used coding agents to implement much of Audio Insight. That did not remove the need for a tight feedback loop; it made the loop more important.\nThe agents could design the threading model, inspect crash logs, add instrumentation, and reason from raw captures. They could build the plugin, but they could not reliably judge how motion felt inside my particular SoundSource setup. At runnable milestones I installed the AU, watched it on the M1 Max, adjusted settings, and returned observations, screenshots, logs, or copied metrics. Those reports led directly to the presentation-lifetime fix, the fractional Spectrogram scroll, dynamic axis labeling, and the queue redesign.\nFor visual and real-time software, “the code is correct” and “the product feels right” are different claims. An instrumented implementation plus a person looking at the actual display proved far more useful than guessing at either one in isolation.\nWhat is open source today The current code identifies itself as Audio Insight 0.1.0. It targets macOS 15 on arm64 and builds AUv2 and VST3. It uses C++20, CMake, a pinned JUCE submodule for the plugin shell, CPU FFT analysis accelerated by Apple\u0026rsquo;s vDSP on macOS, and a native Metal renderer. Project-owned code is licensed under AGPL-3.0-or-later; JUCE retains its own upstream AGPL terms.\nThe current release policy is pragmatic for a small open-source project: builds use ad hoc signing, and Developer ID signing and notarization are out of scope. Users can build from source. For a downloaded bundle, the documented flow is to verify the published checksum, extract it, clear quarantine only on the intended bundle, apply an ad hoc signature, and verify that signature.\nOlder macOS versions, Intel/Universal builds, Windows, and AUv3 are architectural possibilities rather than current support promises. Logic compatibility, broader VST3 host coverage, multi-instance stress testing, and several formal performance gates also remain work in progress.\nThe source, build instructions, and current limitations are all in the Audio Insight repository. If you use analyzers but have never looked inside one, I hope the code makes the path from samples to pixels a little less mysterious. And if the Spectrogram glides across a 120 Hz display without drawing attention to the renderer, that is exactly the point.\n","date":"2026-08-16T16:42:45+08:00","permalink":"https://charlie0129.github.io/blog/p/building-audio-insight/","title":"Building Audio Insight: An Audio Analyzer That Actually Renders Smoothly"},{"content":"I ordered an OVHcloud Kimsufi KS-5 dedicated server for 19.90 USD/month. It is not fast by modern standards, but it is a very usable small Proxmox box if you install it carefully.\nThe goal for this machine was:\nProxmox VE on mirrored ZFS root. Keep only part of each SSD for the root pool. Use the remaining SSD space as a fast disposable scratch pool. Put VMs and CTs behind NAT, because the server only has one usable public IPv4 address and one /128 IPv6 address. Keep host writes low where it is easy to do so. This post is written from the actual KS-5 I installed. The finished machine is running Proxmox VE 9.2.\nHardware The server I received:\nPart Value Product Kimsufi KS-5 CPU Intel Xeon E3-1270 v6, 4C/8T, 3.8 GHz base, 4.2 GHz turbo RAM 32 GB DDR4 ECC 2400 MHz Storage 2x Intel P3520 450 GB NVMe SSD NIC 2x Intel I210 gigabit The PCI devices looked like this:\n1 2 3 4 5 6 7 8 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers (rev 05) 00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31) 00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31) 02:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 02) 03:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 02) 04:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1) (rev 05) 05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) 06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) The SSDs were not new in power-on hours, but the wear was low and both drives had no media errors:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 # nvme smart-log /dev/nvme0n1 critical_warning : 0 temperature : 23 C available_spare : 98% percentage_used : 6% Data Units Read : 8.41 TB Data Units Written : 95.99 TB power_cycles : 107 power_on_hours : 45628 unsafe_shutdowns : 4 media_errors : 0 num_err_log_entries : 0 # nvme smart-log /dev/nvme1n1 critical_warning : 0 temperature : 23 C available_spare : 98% percentage_used : 7% Data Units Read : 142.27 TB Data Units Written : 112.67 TB power_cycles : 74 power_on_hours : 54153 unsafe_shutdowns : 5 media_errors : 0 num_err_log_entries : 0 For a cheap dedicated server, that is acceptable.\nWhy Install Proxmox Manually OVHcloud can install an OS for you from the control panel. I did not use that path for Proxmox.\nThe problem is storage layout. Proxmox works best when it owns the disks directly and can put ZFS on the raw devices. The OVHcloud installer tends to build layouts around mdadm RAID and regular filesystems, or it uses ZFS in a way that treats VM/CT storage more like ordinary directories. That leaves some Proxmox storage features unavailable or awkward, especially thin-provisioned disks and the normal ZFS-backed workflow.\nSo I installed Proxmox myself.\nFormat NVMe Drives as 4K LBA Before installing the OS, boot the server into OVHcloud rescue mode and check the NVMe namespace formats.\nMy Intel P3520 drives supported both 512-byte and 4096-byte LBA formats:\n1 2 3 4 5 6 7 8 # nvme id-ns -H /dev/nvme0n1 LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use) LBA Format 1 : Metadata Size: 8 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good LBA Format 2 : Metadata Size: 16 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good LBA Format 3 : Metadata Size: 0 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best LBA Format 4 : Metadata Size: 8 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best LBA Format 5 : Metadata Size: 64 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best LBA Format 6 : Metadata Size: 128 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best The drives defaulted to 512-byte LBA. Since these are enterprise SSDs and 4K was available, I reformatted both namespaces to format 3: 4096-byte sectors with no metadata.\nThis erases all data on the drive.\n1 2 nvme format /dev/nvme0n1 -l 3 nvme format /dev/nvme1n1 -l 3 After formatting, nvme list should show 4 KiB + 0 B:\n1 2 3 Node Model Namespace Usage Format FW Rev /dev/nvme0n1 INTEL SSDPE2MX450G7 0x1 450.10 GB / 450.10 GB 4 KiB + 0 B MDV10290 /dev/nvme1n1 INTEL SSDPE2MX450G7 0x1 450.10 GB / 450.10 GB 4 KiB + 0 B MDV10290 Open the IPMI KVM In the OVHcloud console, open the IPMI KVM.\nThe KVM launches through an old Java JNLP applet.\nOne annoying detail: the IP address you use to access the OVHcloud console should be the same public client IP used when opening the IPMI KVM. I hit a failure mode where the KVM was blocked because I accessed them through different egress IPs.\nThe machine uses an Intel server board. The useful hotkeys are:\nF2: BIOS setup. F6: one-time boot menu. The boot logo confirms it is an Intel board:\nYou can mount a virtual ISO from the JViewer client:\nThen use F6 and choose the virtual CD-ROM:\n1 2 3 4 5 6 7 8 9 10 Please select boot device: UEFI IPv4: Intel I210 Network 00 at Baseboard UEFI IPv4: Intel I210 Network 00 at Baseboard 2 UEFI IPv6: Intel I210 Network 00 at Baseboard UEFI IPv6: Intel I210 Network 00 at Baseboard 2 Launch EFI Shell Enter Setup UEFI Virtual CDROM 1.00 # Choose this one! UEFI Misc Device Do Not Stream the Proxmox ISO Through JViewer My first attempt was to mount the Proxmox installer ISO directly in JViewer and boot it.\nThat technically works, but it was unusably slow. Even when I ran JViewer from an OVH server (which should have enough bandwidth), virtual media throughput was capped at around 64 KB/s. Installing Proxmox by streaming a full ISO through that path would take forever.\nThe fix is to boot a tiny netboot image first.\nBoot Proxmox Through netboot.xyz Download the netboot.xyz UEFI ISO, mount that ISO in JViewer, and boot it. The image is tiny, so the slow virtual media path is no longer a problem.\nIn netboot.xyz, choose:\n1 2 3 Linux Network Installs (64-bit) Proxmox Proxmox VE Text Installer You can choose the debug installer if you want shell access between install stages. That is useful when something goes wrong.\nnetboot.xyz downloads the Proxmox installer over the server\u0026rsquo;s own network connection and then boots it. OVHcloud provides DHCP even for dedicated servers, so the installer had network access without manual IP configuration.\nOne caveat: after netboot.xyz hands off to the Proxmox installer, the installer does not use a serial console, so Serial over LAN will not work. You still need the JViewer window to see and control the installation.\nInstall Proxmox on ZFS RAID1 In the Proxmox installer, choose ZFS RAID1 across both NVMe drives.\nI intentionally did not give the full SSDs to rpool. The installer has an hdsize option in advanced storage options. I set it to 128 GiB so the root pool would be a 2-way mirror, leaving the remaining space on both SSDs unused for a later scratch pool.\nThis is the important part: decide this during installation. Growing ZFS into extra space is easy. Shrinking an existing ZFS pool is not.\nThe final root layout on my machine:\n1 2 3 4 5 6 7 8 9 10 11 nvme0n1 419.2G disk |-nvme0n1p1 1000K part |-nvme0n1p2 1G part vfat |-nvme0n1p3 128G part zfs_member `-nvme0n1p4 290.2G part zfs_member nvme1n1 419.2G disk |-nvme1n1p1 1000K part |-nvme1n1p2 1G part vfat |-nvme1n1p3 128G part zfs_member `-nvme1n1p4 290.2G part zfs_member p3 on both drives is the mirrored root pool:\n1 2 3 4 5 6 7 8 9 pool: rpool state: ONLINE config: NAME STATE rpool ONLINE mirror-0 ONLINE nvme-INTEL_SSDPE2MX450G7_CVPF71620037450RGN-part3 ONLINE nvme-INTEL_SSDPE2MX450G7_CVPF721600N5450RGN-part3 ONLINE p4 on both drives is the later scratch pool.\nOptional Partition Alignment Fix The Proxmox installer did not give me a perfectly round 1 MiB-aligned final size for the ZFS partition (nvmeXn1p3). If you care about this, fix it immediately after installation while the layout is still simple.\nThe safe method is:\nUse fdisk /dev/nvmeXn1 List partitions with p and note the start/end sector of part 3. Delete d part 3 Create a new part 3 with the same start sector and a new end sector. Increase the end sector a bit, align end sector - 1 to 1M (256 4k sectors). Do not wipe the ZFS signature. Write changes with w. Ask ZFS to expand into the recreated partition. Example:\n1 zpool online -e rpool \u0026lt;device\u0026gt; Note that if you expanded the ZFS part too little (smaller than metaslabs size, typically 1GiB), ZFS may not actually expand the pool to use the empty space. This is fine.\nFind the exact \u0026lt;device\u0026gt; with:\n1 zpool status -v rpool This is not required for a working system. I did it because I wanted the 128 GiB root partition to be exact and aligned. So later partition 4 will not be misaligned either.\nFix OVHcloud Boot-to-Disk After a manual install, the server may not boot straight into Proxmox even though the installation succeeded.\nOVHcloud bare-metal boot is not just \u0026ldquo;BIOS loads local disk\u0026rdquo;. The normal path is roughly:\nThe server PXE-boots from the public interface. OVHcloud DHCP gives the server its public IP and an iPXE loader. iPXE queries OVHcloud\u0026rsquo;s internal boot service. The boot service returns a script based on your configured boot mode. In boot-to-disk mode, iPXE uses sanboot with an EFI bootloader path. If you installed through the OVHcloud panel, OVHcloud knows the EFI bootloader path. If you installed manually, that path can be missing or wrong.\nWhen sanboot fails, iPXE falls back to rEFInd. rEFInd scans the EFI system partitions and may choose the wrong bootloader. In my case it picked memtest first:\n1 2 3 4 rEFInd - Booting OS Starting memtest86+x64.efi Using load options \u0026#39;\u0026#39; The rEFInd menu showed both memtest and systemd-boot:\n1 2 3 4 Boot memtest86+x64.efi from 1021 MiB FAT volume Boot EFI\\systemd\\systemd-bootx64.efi from 1021 MiB FAT volume Boot memtest86+x64.efi from 1021 MiB FAT volume Boot EFI\\systemd\\systemd-bootx64.efi from 1021 MiB FAT volume For Proxmox installed with UEFI and root-on-ZFS, the bootloader is systemd-boot:\n1 \\EFI\\systemd\\systemd-bootx64.efi Set that path through the OVHcloud CLI:\n1 2 ovhcloud login ovhcloud baremetal edit nsXXX.ip-X-X-X.eu --efi-bootloader-path \u0026#39;\\EFI\\systemd\\systemd-bootx64.efi\u0026#39; After that, boot-to-disk should go directly into Proxmox instead of falling through to rEFInd.\nBasic Post-Install ZFS Settings Enable TRIM on the root pool and set basic dataset properties:\n1 2 3 4 5 6 zpool set autotrim=on rpool zfs set atime=off rpool # The default \u0026#39;on\u0026#39; is also lz4. Just set it to lz4 explicitly to be sure. zfs set compression=lz4 rpool # 128k is the default (this is good for HDDs). But 16-32k is better for NVMe array and workloads with small files, which is the case for most VMs/CTs and Proxmox OS. I am using 2-way mirror so I don\u0026#39;t need to worry about 4k ashift adding up. zfs set recordsize=16k rpool recordsize affects file datasets, not zvol block devices. For VM disks backed by ZFS zvols, check the Proxmox storage block size instead. On new Proxmox installations the default is already 16K; older installations may still use 8K. You can check or change it in:\n1 Datacenter -\u0026gt; Storage -\u0026gt; local-zfs -\u0026gt; Edit -\u0026gt; Block Size I also use a few ZFS module options:\n1 2 3 4 5 6 7 8 9 10 # You may remove zfs_arc_max set by the installer if you want to maximize ZFS ARC. Note that ARC may not shrink fast enough under memory pressure, so keep this in mind. cat \u0026gt;/etc/modprobe.d/zfs.conf \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; options zfs zfs_txg_timeout=30 options zfs zfs_trim_txg_batch=128 options zfs zfs_dirty_data_sync_percent=80 options zfs zfs_delay_min_dirty_percent=95 EOF update-initramfs -u -k all The intent:\nzfs_txg_timeout=30: reduce idle write frequency. zfs_trim_txg_batch=128: make TRIM batching less tiny for NVMe. zfs_dirty_data_sync_percent=80 and zfs_delay_min_dirty_percent=95: delay throttling until the dirty-data situation is actually serious. These are not universal defaults. They fit this small host because the workload is mostly personal VMs/CTs and scratch tasks, not a database with strict latency guarantees.\nReduce Unnecessary Host Writes I am not using Proxmox clustering on this server, so I disabled the cluster HA services:\n1 2 3 systemctl disable --now pve-ha-crm.service systemctl disable --now pve-ha-lrm.service systemctl disable --now corosync.service Limit persistent journald usage:\n1 2 sed -i \u0026#39;s/.*SystemMaxUse.*/SystemMaxUse=128M/g\u0026#39; /etc/systemd/journald.conf systemctl restart systemd-journald Configure BBR Use BBR with fq.\nOn Debian/Proxmox, a later default sysctl file can set net.core.default_qdisc=fq_codel after your file during boot. Use a late filename:\n1 2 3 4 5 6 cat \u0026gt;/etc/sysctl.d/99-bbr.conf \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; net.core.default_qdisc = fq net.ipv4.tcp_congestion_control = bbr EOF sysctl --system Check it:\n1 sysctl net.core.default_qdisc net.ipv4.tcp_congestion_control Expected:\n1 2 net.core.default_qdisc = fq net.ipv4.tcp_congestion_control = bbr Configure Swap Always have swap when using ZFS, even with enough RAM. It gives the kernel somewhere to go if ARC does not shrink quickly enough under memory pressure, and it reduces the chance that the OOM killer targets useful applications.\nI use ZRAM swap:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 git clone --depth=1 https://github.com/foundObjects/zram-swap.git cd zram-swap ./install.sh cd .. rm -rf zram-swap cat \u0026gt;/etc/sysctl.d/zram.conf \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; vm.swappiness = 180 vm.watermark_boost_factor = 0 vm.watermark_scale_factor = 125 vm.page-cluster = 0 EOF sysctl --system You may use disk swap if you want.\nMy current host has a 38.8 GiB ZRAM swap device:\n1 2 NAME TYPE SIZE USED PRIO /dev/zram0 partition 38.8G 0B 15 SSH and Serial Console I like SSH sessions to die quickly when the client disappears unexpectedly:\n1 2 ClientAliveInterval 60 ClientAliveCountMax 3 Put that in sshd_config or a file under sshd_config.d, then restart SSH.\nFor emergency access, enable OVHcloud serial-over-LAN by adding a serial console to the Proxmox kernel command line:\n1 2 3 additional_cmdline=\u0026#34;console=tty0 console=ttyS0,115200n8\u0026#34; sed -i \u0026#34;s|$| $additional_cmdline|\u0026#34; /etc/kernel/cmdline proxmox-boot-tool refresh On my host, /etc/kernel/cmdline now contains:\n1 root=ZFS=rpool/ROOT/pve-1 boot=zfs vmlinuz video=vesafb:ywrap,mtrr initrd=initrd.magic console=tty0 console=ttyS0,115200n8 Scratch Storage The root pool uses 128 GiB on each SSD. The remaining 290.2 GiB on each SSD is free for scratch storage.\nI considered two options:\nmdadm RAID0 + LVM thin + ext4/XFS. ZFS striped pool. The mdadm option may win in some microbenchmarks, but the management cost is higher. I used a ZFS striped pool because it is simple and good enough.\nThis pool has no redundancy. If either SSD fails, the scratch pool is gone. That is fine for my use case because it is scratch space.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 zpool create -f \\ -o ashift=12 \\ -o autotrim=on \\ scratchpool \\ /dev/disk/by-id/nvme-INTEL_SSDPE2MX450G7_CVPF721600N5450RGN-part4 \\ /dev/disk/by-id/nvme-INTEL_SSDPE2MX450G7_CVPF71620037450RGN-part4 zfs set recordsize=16k scratchpool zfs set atime=off scratchpool zfs set compression=lz4 scratchpool zfs set logbias=throughput scratchpool zfs create scratchpool/unsafe zfs set sync=disabled scratchpool/unsafe zfs set copies=1 scratchpool/unsafe Current state:\n1 2 3 4 5 6 7 8 pool: scratchpool state: ONLINE config: NAME STATE scratchpool ONLINE nvme-INTEL_SSDPE2MX450G7_CVPF721600N5450RGN-part4 ONLINE nvme-INTEL_SSDPE2MX450G7_CVPF71620037450RGN-part4 ONLINE Properties:\n1 2 3 NAME AVAIL MOUNTPOINT RECSIZE COMPRESS ATIME SYNC COPIES scratchpool 562G /scratchpool 16K lz4 off standard 1 scratchpool/unsafe 562G /scratchpool/unsafe 16K lz4 off disabled 1 Use scratchpool/unsafe only for data you can recreate. sync=disabled lies to applications about sync writes.\nmdadm + LVM Alternative If you want the non-ZFS scratch layout, create one extra partition on each SSD and build RAID0:\n1 2 3 4 5 6 7 8 9 10 11 12 apt install -y mdadm mdadm --create /dev/md0 \\ --verbose \\ --level=0 \\ --raid-devices=2 \\ --chunk=128K \\ /dev/disk/by-id/nvme-INTEL_SSDPE2MX450G7_CVPF721600N5450RGN-part4 \\ /dev/disk/by-id/nvme-INTEL_SSDPE2MX450G7_CVPF71620037450RGN-part4 mdadm --detail --scan \u0026gt;\u0026gt; /etc/mdadm/mdadm.conf update-initramfs -u For alignment:\nmdadm chunk size: 128K. 128K is a good balance between performance and overhead for NVMe drives. If you want to maximize sequential performance, use larger values like 1M. But remember to align filesystem / LVM with this value. Data disks: 2. Stripe width: 256K. LVM data alignment: 1M is fine because it is a multiple of 256K. NVMe (physical) → mdadm RAID0 → LVM PV → LVM LV / thin pool → filesystem\nWhen you create LVM on top of RAID-0, you must ensure:\nLVM PE size = multiple of mdadm chunk size LVM PV starts aligned to mdadm stripe boundary Create the LVM layer:\n1 2 3 4 5 6 7 8 9 10 11 12 13 # The default dataalignment value is 1M, which is a multiple of 256K, so it\u0026#39;s fine. pvcreate --dataalignment 1M /dev/md0 # Default VG extent size is often 4M, which is multiples of 1M data alignment, # which is also multiples of 256K stripe width. Using the default value is fine. vgcreate --physicalextentsize 4M vg_scratch /dev/md0 # Thin pool chunk size defines how much physical space is reserved when any # part of a chunk is touched. It does not defined write size. So it can be # smaller than 256k. The default 64k is fine. I will just match it with the # stripe width = 256k. # The default metadata size can be too big if you are using small driver # like 16G Optane drive. Calculate the desired metadata size (bytes) # by `48 * (total size / chunk size)`. lvcreate -l 100%FREE --thin vg_scratch/scratch_thin --chunksize 256k --poolmetadatasize 512M When formatting a filesystem inside a VM or manually on the host, pass the stripe geometry:\n1 2 3 4 5 6 7 8 9 # ext4 # stride = mdadm chunk size / filesystem block size = 128k / 4k = 32 # stripe-width = stride * disks = 32 * 2 = 64 mkfs.ext4 -b 4096 -E stride=32,stripe-width=64 /dev/vg_scratch/lv # XFS # su = mdadm chunk size = 128k # sw = data disks = 2 mkfs.xfs -d su=128k,sw=2 /dev/vg_scratch/lv Fast but unsafe mount options for scratch data:\n1 2 ext4: noatime,nodiratime,nobarrier,data=writeback,commit=60 xfs: noatime,nodiratime,logbufs=8,logbsize=256k For CTs, Proxmox does not make it convenient to set filesystem mount options on a normal CT mountpoint. If you need custom options, create the LV manually, mount it on the host, and bind-mount it into the CT:\n1 2 3 4 5 6 7 8 lvcreate -V 16G -T vg_scratch/scratch_thin -n scratch_test mkfs.xfs -d su=128k,sw=2 /dev/vg_scratch/scratch_test mkdir -p /mnt/scratch_test mount -o noatime,nodiratime,logbufs=8,logbsize=256k /dev/vg_scratch/scratch_test /mnt/scratch_test chmod 777 /mnt/scratch_test pct set 199 -mp0 /mnt/scratch_test,mp=/mnt/scratch_test If your workload is random-write heavy and can fit on one SSD, skip RAID0 and LVM entirely. A single raw disk or single partition is often better for small random writes than a layered RAID0 + LVM-thin setup.\nUsing Scratch Storage in VMs and CTs For VMs, add a virtual disk on the scratch pool. Enable discard and SSD emulation on the virtual disk. Inside the guest, enable periodic TRIM. When formatting the disk, make sure the filesystem is aligned to the underlying ZFS volsize or LVM stripe geometry. For example, if underlying ZFS volsize is 16K, use mkfs.ext4 -O bigalloc -C 16384 /dev/yourdisk. If the underlying storage is LVM+RAID, use the correct stride and stripe-width options when formatting.\nFor CTs, add a mountpoint on the scratch pool. ZFS mountpoints are automatically aligned. For LVM+RAID, make sure the filesystem is aligned to the underlying geometry. TRIM is handled by the host for ZFS mountpoints. For LVM+RAID, run pct fstrim on the CT.\nTo use the scratch pool for temporary data, you can use bind-mounts to mount common scratch directories to scratch disk mountpoints. This avoids changing the config for each applications such as Docker, containerd and etc.\nSuppose the scratch disk is mounted at /mnt/unsafe, and you want to use it for Docker, containerd, and ~/.cache. Write the following lines in /etc/fstab:\n1 2 3 /mnt/unsafe/docker /var/lib/docker none rbind 0 0 /mnt/unsafe/containerd /var/lib/containerd none rbind 0 0 /mnt/unsafe/.cache /root/.cache none rbind 0 0 1 2 3 4 mkdir -p /mnt/unsafe/{docker,containerd,.cache} rm -rf /var/lib/docker /var/lib/containerd /root/.cache mkdir -p /var/lib/docker /var/lib/containerd /root/.cache mount -av TRIM For ZFS pools:\n1 2 zpool set autotrim=on rpool zpool set autotrim=on scratchpool For VMs, enable discard and SSD emulation on virtual disks, then enable periodic TRIM inside the guest.\nFor CTs backed by ZFS mountpoints, the host handles it. For CTs backed by other storage such as LVM thin, run pct fstrim:\n1 2 3 4 5 6 7 8 9 cat \u0026gt;/usr/local/bin/trimcts.sh \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; #!/bin/bash for id in $(pct list | awk \u0026#39;$2 == \u0026#34;running\u0026#34; {print $1}\u0026#39;); do echo \u0026#34;==\u0026gt; Trimming CT $id\u0026#34; pct fstrim \u0026#34;$id\u0026#34; done EOF chmod +x /usr/local/bin/trimcts.sh Then run it from cron or a systemd timer. For example, weekly from cron:\n1 0 2 * * 7 /usr/local/bin/trimcts.sh \u0026gt;/var/log/trimcts.log 2\u0026gt;\u0026amp;1 Proxmox Networking on a Single Public IP Proxmox creates a bridge by default and attaches the physical NIC to it. That is usually the right setup when the server is in your own LAN.\nIt is not right for this Kimsufi server.\nThe server has one usable public IPv4 address and one /128 IPv6 address. It does not have a routed public subnet for VMs and CTs. If VMs are bridged directly to the public interface, they do not get usable public addresses.\nSo I do this instead:\nPut the public IPv4 and public IPv6 directly on eno1, not vmbr0. By default, Proxmox puts the public IP on the vmbr0 bridge, which is not what we want. You should move it to the physical interface. Create vmbr0 as an internal bridge with no physical ports. Put VMs and CTs on vmbr0. NAT outbound traffic from vmbr0 to eno1. Port-forward selected inbound ports to service CTs. Current /etc/network/interfaces:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 auto lo iface lo inet loopback auto eno1 iface eno1 inet static address \u0026lt;public-ipv4\u0026gt;/24 gateway \u0026lt;public-ipv4-gateway\u0026gt; iface eno1 inet6 static address \u0026lt;public-ipv6\u0026gt;/128 gateway \u0026lt;public-ipv6-gateway\u0026gt; iface eno2 inet manual auto vmbr0 iface vmbr0 inet static address 10.187.54.1/24 bridge-ports none bridge-stp off bridge-fd 0 iface vmbr0 inet6 static address fc10:187:54::1/64 source /etc/network/interfaces.d/* Enable forwarding:\n1 2 3 4 5 6 cat \u0026gt;/etc/sysctl.d/99-forward.conf \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; net.ipv4.ip_forward = 1 net.ipv6.conf.all.forwarding = 1 EOF sysctl --system About NAT66: it is usually the wrong IPv6 design. If you have a routed IPv6 prefix, route it to your VM bridge. On this server I only have one /128, so NAT66 is the practical workaround if VMs/CTs need outbound IPv6.\nNAT and Port Forwarding NAT rewrites packet addresses as traffic crosses the host.\nFor outbound IPv4 NAT, a CT sends a packet like:\n1 src=10.187.54.100:51514 dst=1.1.1.1:443 After MASQUERADE on the host:\n1 src=\u0026lt;public-ipv4\u0026gt;:51514 dst=1.1.1.1:443 Linux conntrack records that translation so the reply can be translated back and forwarded to the CT.\nThere is a Proxmox-specific caveat when the firewall is enabled on a VM or CT network interface (firewall=1). Proxmox places that interface behind an fwbr* firewall bridge. With bridge netfilter enabled, conntrack can record the connection while it crosses the firewall bridge, before it reaches the host\u0026rsquo;s normal NAT path. The later FORWARD rule can accept the packet, but POSTROUTING may reuse the already tracked non-NATed connection instead of attaching MASQUERADE or SNAT. The packet then leaves the public interface with its private source address and never receives a reply.\nPut the firewall-bridge tracking in a separate conntrack zone by adding these hooks to the vmbr0 stanza in /etc/network/interfaces:\n1 2 3 4 post-up iptables -t raw -C PREROUTING -i fwbr+ -j CT --zone 1 2\u0026gt;/dev/null || iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1 2\u0026gt;/dev/null || true post-up ip6tables -t raw -C PREROUTING -i fwbr+ -j CT --zone 1 2\u0026gt;/dev/null || ip6tables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1 post-down ip6tables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1 2\u0026gt;/dev/null || true The -C/|| guard matters on a dual-stack vmbr0: ifupdown2 can run the hook once for each address family.\nThe IPv4 rules cover MASQUERADE/SNAT, while the IPv6 rules are also needed when using NAT66. A characteristic symptom is that NAT works for a guest without the Proxmox interface firewall but fails for an otherwise identical firewall-enabled guest. In conntrack output, the broken flow\u0026rsquo;s reply destination remains the guest\u0026rsquo;s private address instead of the host\u0026rsquo;s public address. Allowing all OUT and FORWARD traffic does not fix it because this is a conntrack/NAT-path problem, not a filter-policy rejection.\nInbound port forwarding is DNAT. A client connects to the public server:\n1 src=198.51.100.20:41000 dst=\u0026lt;public-ipv4\u0026gt;:443 The host rewrites the destination before routing:\n1 src=198.51.100.20:41000 dst=10.187.54.3:443 I use my own small tool, iptfwd, to manage these rules from a config file: charlie0129/iptfwd.\nExample config for this topology:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 defaults: public_iface: eno1 private_iface: vmbr0 manage_filter: true enable_ip_forwarding: true nat: - name: intranet-v4 source: 10.187.54.0/24 type: masquerade - name: intranet-v6 source: fc10:187:54::/64 type: snat to_source: \u0026lt;public-ipv6\u0026gt; rules: - name: service-http-v4 proto: tcp public_port: 80 target: 10.187.54.3 target_port: 80 - name: service-https-v4 proto: tcp public_port: 443 target: 10.187.54.3 target_port: 443 - name: service-http-v6 proto: tcp public_port: 80 target: fc10:187:54::3 target_port: 80 - name: service-https-v6 proto: tcp public_port: 443 target: fc10:187:54::3 target_port: 443 For IPv4, MASQUERADE is convenient. For NAT66, I prefer explicit SNAT to the public IPv6 address. IPv6 interfaces can have multiple addresses, and explicit SNAT makes the translated source predictable.\nIf you do not want to use iptfwd, the equivalent basic iptables rules are:\n1 2 3 4 5 6 7 8 9 10 11 iptables -t nat -A POSTROUTING -s 10.187.54.0/24 -o eno1 -j MASQUERADE iptables -A FORWARD -i vmbr0 -o eno1 -j ACCEPT iptables -A FORWARD -i eno1 -o vmbr0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT public_ipv6=\u0026lt;YOUR_PUBLIC_IPV6\u0026gt; ip6tables -t nat -A POSTROUTING -s fc10:187:54::/64 -o eno1 -j SNAT --to-source \u0026#34;$public_ipv6\u0026#34; ip6tables -A FORWARD -i vmbr0 -o eno1 -j ACCEPT ip6tables -A FORWARD -i eno1 -o vmbr0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT apt install iptables-persistent netfilter-persistent save Before setting up DHCP, test NAT manually:\nCreate a CT on vmbr0. Set IPv4 to something like 10.187.54.100/24. Set IPv4 gateway to 10.187.54.1. Set IPv6 to something like fc10:187:54::100/64. Set IPv6 gateway to fc10:187:54::1. Test: 1 2 ping www.google.com ping6 www.google.com If that fails, debug NAT before adding DHCP.\nDHCP and DNS with Pi-hole I use Pi-hole as DHCP and DNS for the private bridge because it has a nice web UI and makes static leases easy.\nCreate a small Alpine CT:\nBridge: vmbr0. IPv4: 10.187.54.2/24. IPv4 gateway: 10.187.54.1. IPv6: fc10:187:54::2/64. IPv6 gateway: fc10:187:54::1. CPU: 1 core is enough. RAM: 128 MB is enough. Disk: 512 MB is enough. Install Pi-hole:\n1 2 3 4 apk update apk add curl bash curl -sSL https://install.pi-hole.net | bash # You may disable query logs to reduce writes to the database. If you forget the web password:\n1 pihole setpassword Reduce query-log database writes:\n1 pihole-FTL --config database.DBinterval 1800 In the Pi-hole UI:\nEnable DHCP for 10.187.54.0/24. Set the router/gateway to 10.187.54.1. Configure DNS as desired. Disable NTP sync under Settings -\u0026gt; All Settings -\u0026gt; Network Time Sync. Uncheck ntp.ipv4/ipv6/sync.active. An unprivileged CT cannot set system time anyway. Alpine log cleanup I use for small CTs:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 cat \u0026gt;/etc/periodic/hourly/logtruncate \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; #!/bin/sh LOG_DIR=\u0026#34;/var/log\u0026#34; MAX_SIZE=$((4 * 1024 * 1024)) find \u0026#34;$LOG_DIR\u0026#34; -type f ! -name \u0026#39;*.0\u0026#39; | while read -r file; do size=$(stat -c %s \u0026#34;$file\u0026#34; 2\u0026gt;/dev/null || stat -f %z \u0026#34;$file\u0026#34; 2\u0026gt;/dev/null || echo 0) if [ \u0026#34;$size\u0026#34; -gt \u0026#34;$MAX_SIZE\u0026#34; ]; then logger -t logrotate \u0026#34;Rotating $file ($size bytes)\u0026#34; cp \u0026#34;$file\u0026#34; \u0026#34;$file.0\u0026#34; truncate -s 0 \u0026#34;$file\u0026#34; logger -t logrotate \u0026#34;Rotated $file\u0026#34; fi done EOF chmod +x /etc/periodic/hourly/logtruncate sed -i \u0026#39;s/^SYSLOGD_OPTS=.*/SYSLOGD_OPTS=\u0026#34;-t -s 0\u0026#34;/\u0026#39; /etc/conf.d/syslog cat \u0026gt;/etc/periodic/daily/apkcacheclean \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; #!/bin/sh find /var/cache/apk -type f -name \u0026#39;*.apk\u0026#39; -mtime +7 -delete find /var/cache/apk -type f -name \u0026#39;*.tar.gz\u0026#39; -mtime +90 -delete EOF chmod +x /etc/periodic/daily/apkcacheclean Host Monitoring I keep monitoring tools in Podman containers instead of installing everything directly on the Proxmox host.\nInstall Podman:\n1 2 3 4 5 6 # So Intel PCM can work. echo msr \u0026gt;\u0026gt; /etc/modules modprobe msr # Install podman to run monitoring tools in containers. Do not include any unnecessary packages to keep the host clean and minimal. Remember to install aardvark-dns to make container DNS work. apt install --no-install-recommends podman aardvark-dns On current Debian/Proxmox, Podman should use overlay storage on ZFS:\n1 podman info --format \u0026#39;{{.Store.GraphDriverName}}\u0026#39; Expected:\n1 overlay Make podman-restart.service also handle containers with restart-policy=unless-stopped:\n1 2 3 4 5 6 7 8 9 10 mkdir -p /etc/systemd/system/podman-restart.service.d cat \u0026gt;/etc/systemd/system/podman-restart.service.d/override.conf \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; [Service] ExecStart=/usr/bin/podman $LOGGING start --all --filter restart-policy=always --filter restart-policy=unless-stopped ExecStop=/usr/bin/podman $LOGGING stop --all --filter restart-policy=always --filter restart-policy=unless-stopped EOF systemctl daemon-reload systemctl enable podman-restart.service Enable the Podman socket and Docker-compatible CLI:\n1 2 3 4 5 systemctl enable --now podman.socket apt install --no-install-recommends docker-cli docker-compose docker context create podman --docker \u0026#34;host=unix:///run/podman/podman.sock\u0026#34; docker context use podman An example monitoring compose file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 services: vmagent: container_name: vmagent image: victoriametrics/vmagent:v1.144.0 restart: unless-stopped mem_limit: 512M environment: GOMEMLIMIT: 900MiB command: - -httpListenAddr=0.0.0.0:8428 - -cacheExpireDuration=5m - -promscrape.config=/etc/vmagent/scrape.yml - -remoteWrite.url=https://XXX # Your VictoriaMetrics remote write endpoint - -remoteWrite.basicAuth.username=XXX - -remoteWrite.basicAuth.password=XXX - -remoteWrite.maxDiskUsagePerURL=128MiB - -remoteWrite.tmpDataPath=/var/vmagent - -remoteWrite.vmProtoCompressLevel=3 - -remoteWrite.flushInterval=66s - -remoteWrite.label=instance=XXX extra_hosts: - \u0026#34;host.docker.internal:host-gateway\u0026#34; volumes: - ./vmagent/config:/etc/vmagent - ./vmagent/data:/var/vmagent smartctlexporter: container_name: smartctlexporter image: prometheuscommunity/smartctl-exporter:v0.14.0 restart: unless-stopped mem_limit: 64M user: 0:0 environment: GOMEMLIMIT: 110MiB privileged: true nodeexporter: container_name: nodeexporter image: prom/node-exporter:v1.11.1 restart: unless-stopped mem_limit: 64M environment: GOMEMLIMIT: 110MiB privileged: true command: - --path.rootfs=/host - --web.listen-address=:9100 network_mode: host pid: host volumes: - \u0026#34;/:/host:ro,rslave\u0026#34; podmanexporter: container_name: podmanexporter image: quay.io/navidys/prometheus-podman-exporter:v1.21.0 mem_limit: 64M user: 0:0 privileged: true environment: GOMEMLIMIT: 110MiB CONTAINER_HOST: unix:///run/podman/podman.sock restart: unless-stopped volumes: - /run/podman/podman.sock:/run/podman/podman.sock command: - --web.listen-address=:9882 cgroupexporter: container_name: cgroupexporter image: ghcr.io/arianvp/cgroup-exporter:0.3.3-amd64 mem_limit: 64M environment: GOMEMLIMIT: 110MiB restart: unless-stopped volumes: - /sys/fs/cgroup:/sys/fs/cgroup:ro command: - -listen-address=:3232 pcm: container_name: pcm image: opcm/pcm mem_limit: 96M restart: unless-stopped privileged: true Service CT I keep public-facing services in a separate CT and forward only selected ports from the host.\nCreate an unprivileged Alpine CT on vmbr0, then:\n1 2 3 apk add openssh openssh-server rc-update add sshd default rc-service sshd start Add this to /etc/ssh/sshd_config:\n1 2 3 PermitRootLogin yes ClientAliveInterval 60 ClientAliveCountMax 3 Install Podman inside the CT:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 cat \u0026gt;/usr/local/bin/enablecgroup2nesting \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; #!/bin/sh if [ -f /sys/fs/cgroup/cgroup.controllers ]; then echo \u0026#34;Enabling cgroup v2 nesting\u0026#34; mkdir -p /sys/fs/cgroup/init xargs -rn1 \u0026lt; /sys/fs/cgroup/cgroup.procs \u0026gt; /sys/fs/cgroup/init/cgroup.procs || : sed -e \u0026#39;s/ / +/g\u0026#39; -e \u0026#39;s/^/+/\u0026#39; \u0026lt; /sys/fs/cgroup/cgroup.controllers \\ \u0026gt; /sys/fs/cgroup/cgroup.subtree_control fi EOF chmod +x /usr/local/bin/enablecgroup2nesting cat \u0026gt;/etc/init.d/cgroup2nesting \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; #!/sbin/openrc-run description=\u0026#34;Enable nesting of cgroup2.\u0026#34; depend() { keyword -docker -podman -prefix -systemd-nspawn -vserver -wsl after sysfs } start() { ebegin \u0026#34;Enabling cgroup v2 nesting\u0026#34; /usr/local/bin/enablecgroup2nesting eend $? } EOF chmod +x /etc/init.d/cgroup2nesting rc-update add cgroup2nesting default rc-service cgroup2nesting start apk add podman iptables sed -i \u0026#39;s/^#log_size_max =.*/log_size_max = 1048576/\u0026#39; /etc/containers/containers.conf rc-update add podman rc-service podman start apk add docker-cli docker-cli-compose docker context create podman --docker \u0026#34;host=unix:///run/podman/podman.sock\u0026#34; docker context use podman Also check out the Pi-Hole setup section for Alpine log cleanup and other small CT optimizations. You must do this because Alpine does not have log rotation by default. You disk will fill up if you do not manage logs.\nThen run Caddy or Traefik in that CT for reverse proxy and TLS termination. Forward 80 and 443 from the host to the service CT with iptfwd or iptables.\nIf you use a privileged CT with Podman over ZFS, you may need an overlay mount helper:\n1 2 3 4 5 6 7 8 9 10 cat \u0026gt;/usr/local/bin/zfsoverlaymount \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; #!/bin/sh /bin/mount -t overlay overlay \u0026#34;$@\u0026#34; EOF chmod +x /usr/local/bin/zfsoverlaymount cat \u0026gt;/etc/containers/storage.conf \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; [storage.options] mount_program = \u0026#34;/usr/local/bin/zfsoverlaymount\u0026#34; EOF I prefer unprivileged CTs unless there is a specific reason not to use them.\nSMTP Notifications By default, Proxmox may send mail directly and anonymously to the email address you entered during installation. Some providers reject that mail.\nConfigure a real SMTP target instead:\n1 Datacenter -\u0026gt; Notifications -\u0026gt; Notification Targets -\u0026gt; Add -\u0026gt; SMTP Final State The host ended up with:\nProxmox VE 9.2. Mirrored ZFS root pool on 128 GiB from each NVMe SSD. Striped ZFS scratch pool on the remaining SSD space. 4K LBA on both NVMe drives. Private-only vmbr0 for VMs and CTs. IPv4 NAT and IPv6 NAT66 from vmbr0 to eno1. 80/443 forwarded to a service CT. Pi-hole handling private DHCP and DNS. Podman-based host monitoring. For 19.90 USD/month, this is a surprisingly useful little virtualization box.\n","date":"2026-07-06T14:30:00+08:00","permalink":"https://charlie0129.github.io/blog/p/kimsufi-ks5-proxmox/","title":"Installing Proxmox VE on an OVHcloud Kimsufi KS-5"},{"content":"I sometimes need a tiny cloud instance that only forwards traffic. CPU and memory can be small, and the disk is mostly wasted space. On one Alibaba Cloud ecs.t6-c4m1.large instance, a 1 GB root disk on a long contract can push the cost down to about $0.40/month, but a normal Linux install leaves too little usable space.\nThe trick is not complicated:\nUse Alpine Linux, because the base system is small. Keep /boot tiny. A 512 MB boot partition is absurd on a 1 GB disk. Put / on Btrfs with transparent compression. Application files, package metadata, and logs usually compress well. Do not allocate disk swap. Use zram later if the machine needs swap-like behavior. This post is the install recipe I use to build a minimal BIOS/MBR Alpine image, then optionally convert it to QCOW2 for reuse.\nIf you would rather not do this by hand, I later automated the whole thing: Building Alpine Linux Disk Images Without a VM is a script that builds the same class of image as a file, with no VM and no interactive steps, and boots under both BIOS and UEFI. This post is still the explanation of why the pieces are the way they are.\nAssumptions This guide targets a very specific VM shape:\nLegacy BIOS boot with MBR and Syslinux/Extlinux. One small ext4 /boot partition and one Btrfs root partition. No disk encryption, no LVM, no UEFI, no separate /var. The whole disk can be destroyed. If your VM boots with UEFI, use an EFI system partition and GRUB instead. If your disk is NVMe, adapt the partition names to /dev/nvme0n1p1 and /dev/nvme0n1p2.\nThe final layout is:\nPartition Size Filesystem Mountpoint /dev/vda1 64 MB ext4 /boot /dev/vda2 Rest of disk (~500MB is fine) Btrfs / 64 MB is enough for this single-kernel Alpine image. The boot partition usage should be around 30M with Alpine 3.23 virt kernel (6.18). If you plan to keep multiple kernels or use a larger bootloader setup, use 128 MB instead.\nBefore you start installing, create a VM with a ~512MB disk (yes, really). CPU and RAM can be small, because the install process is not resource-intensive. After the install, you can resize the disk to 1 GB or larger.\nAlways go with a small disk first because the disk image can be expanded easily, but shrinking a disk is not trivial.\nPartition The Disk Set the disk variables so the commands below are harder to mistype:\n1 2 3 DISK=/dev/vda BOOT=/dev/vda1 ROOT=/dev/vda2 Now create a DOS/MBR partition table with a tiny boot partition:\n1 fdisk \u0026#34;$DISK\u0026#34; Inside fdisk, enter:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 o # new empty DOS partition table n # new partition p # primary 1 # partition number 2048 # first sector, 1 MiB aligned +64M # size a # toggle bootable flag 1 n # new partition p # primary 2 # partition number 133120 # first sector immediately after the 64 MB partition # press Enter for the default end sector p # verify the table w # write changes The important details are:\n/dev/vda1 should be bootable. /dev/vda1 should start at sector 2048. /dev/vda2 should start at sector 133120. The second partition should use the rest of the disk. Check that the kernel sees the new partitions:\n1 ls -l /dev/vda* If /dev/vda1 and /dev/vda2 do not appear, reboot the live ISO. Alpine\u0026rsquo;s manual disk setup docs also recommend rebooting after manual partition creation when needed.1\nStart Alpine Setup Download the latest Alpine ISO from https://alpinelinux.org/downloads/. I usually use the x86_64 architecture and the virtual variant. As this will be used in virtualized environments, the virt kernel is smaller.\nBoot the Alpine ISO and log in as root. The live environment has no root password by default.\nRun the normal installer first:2\n1 setup-alpine Answer the usual questions for keyboard, hostname, network, DNS, root password, timezone, mirror, SSH, and NTP. I usually choose chrony for NTP.\nWhen it asks which disk to use, type:\n1 none This keeps the base configuration but skips automatic partitioning and formatting. For the later diskless-mode prompts, also choose none for configs. You can keep the default apk cache dir (/var/cache/apk).\nFormat And Mount Install the filesystem tools in the live environment:\n1 apk add btrfs-progs e2fsprogs Format /boot as ext4 and / as Btrfs:\n1 2 mkfs.ext4 -m 0 -L boot \u0026#34;$BOOT\u0026#34; mkfs.btrfs -f -L alpine-root \u0026#34;$ROOT\u0026#34; Mount root with compression enabled:\n1 2 3 4 5 6 # adjust the compression level as desired. zstd:1 is fast and has reasonable compression. zstd:3 is slower but compresses better. Any higher level is usually overkill and comes with diminishing returns. mount -t btrfs -o rw,relatime,compress=zstd:3,ssd,discard=async,space_cache=v2 \u0026#34;$ROOT\u0026#34; /mnt btrfs property set /mnt compression zstd:3 mkdir -p /mnt/boot mount -t ext4 \u0026#34;$BOOT\u0026#34; /mnt/boot I keep /boot on ext4 because it is small and predictable with this Syslinux/MBR setup. On a tiny image, boring boot is good boot.\nInstall Alpine Install Alpine into the mounted filesystem:\n1 setup-disk -v -m sys -s 0 /mnt The options mean:\n-m sys: install a traditional persistent system to disk. -s 0: do not create disk swap. /mnt: use the partitions already mounted under /mnt. This is the standard setup-disk -m sys /mnt flow, just with a manually prepared root filesystem.3\nVerify fstab And initramfs Check /mnt/etc/fstab before rebooting:\n1 vi /mnt/etc/fstab The root entry should include Btrfs compression. Mine looks like this:\n1 2 UUID=... / btrfs rw,relatime,compress=zstd:3,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/ 0 1 UUID=... /boot ext4 rw,relatime 0 2 Remove any stale cdrom or usbdisk entries if the installer generated them.\nAlso check that the installed initramfs knows about Btrfs:\n1 grep -w btrfs /mnt/etc/mkinitfs/mkinitfs.conf If btrfs is missing from features=\u0026quot;...\u0026quot;, add it before rebooting and regenerate the initramfs from a chroot. Alpine explicitly calls this out for manual Btrfs root installs.4\nFix The MBR Bootloader On these tiny MBR images, I have seen the first-stage bootloader not get written correctly. Reinstalling the Syslinux MBR is cheap insurance:\n1 2 apk add syslinux dd bs=440 count=1 conv=notrunc if=/usr/share/syslinux/mbr.bin of=\u0026#34;$DISK\u0026#34; Use gptmbr.bin instead only if you are using GPT. For the DOS partition table above, mbr.bin is the right file.5\nNow unmount and reboot:\n1 2 3 4 5 cd / sync umount /mnt/boot umount /mnt reboot After the first boot, log in and check the actual space usage:\n1 2 df -h / btrfs filesystem usage / Convert The Disk To QCOW2 Once the image is configured the way you want, shut it down and convert the raw disk from the host:\n1 qemu-img convert -f raw -c -p -O qcow2 /dev/mapper/ex950-vm--300--disk--0 alpine-minimal.qcow2 Replace the source device with your VM disk. Do this from a stopped VM or a consistent snapshot, not from a running writable system.\nYou can now reuse the QCOW2 as a base image for other VMs or cloud instances.\nRestore From The Provider\u0026rsquo;s Original OS Some cloud providers do not let you upload or boot a custom disk image. If you still have VNC/serial-console access, another way is to boot the provider\u0026rsquo;s original Linux image, download your Alpine QCOW2 there, convert it to raw, then reboot into the original OS initramfs and overwrite the whole disk from there.\nThe important reason for the two-stage flow is tooling: the original OS probably has curl and qemu-img, while the initramfs usually does not. The initramfs is only used for the final dd, because at that point the real root filesystem is not mounted and can be safely overwritten.\nThis is destructive. Double-check the target disk and the old root partition before running dd.\nFirst, in the provider\u0026rsquo;s original OS. This filesystem needs enough free space to hold both the downloaded QCOW2 and the converted raw image:\n1 2 3 4 5 # Download the Alpine minimal image in qcow2 format and convert it to raw. # Do this in the original OS because the initramfs probably does not have # curl or qemu-img. curl -L \u0026#34;\u0026lt;qcow2 link\u0026gt;\u0026#34; -o /os.qcow2 qemu-img convert -f qcow2 -O raw /os.qcow2 /os.raw Then use the provider console or VNC to reboot into initramfs. On a GRUB-based original OS, select the normal boot entry, press e, find the Linux kernel command line, append:\n1 break=premount Then press F10 or Ctrl-x to boot. This should drop you into an initramfs shell before the root filesystem is mounted.\nFrom the initramfs shell:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 # Find the old root partition and the target disk. # Example: # old root partition: /dev/sda2 # target disk: /dev/sda cat /proc/partitions mkdir /tmp/rootfs cd /tmp # Mount the old root filesystem read-only to copy the prepared image. # Replace /dev/sda2 with the root partition from the original OS, and # replace ext4 if the provider image uses another filesystem. mount -t ext4 -o ro /dev/sda2 rootfs # Copy the raw image to initramfs memory so the old root can be unmounted # before overwriting the disk. # # This requires enough RAM for /os.raw. If the VM is too small, attach a # temporary rescue disk or use a rescue system that can stream the image # from the network instead. dd if=rootfs/os.raw of=os.raw umount rootfs # This overwrites the whole disk. Replace /dev/sda with the target disk. dd if=os.raw of=/dev/sda reboot After booting into Alpine, apply provider-specific network settings. For a static IPv4 setup:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 cat \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; \u0026gt; /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet static address X.X.X.X/24 gateway X.X.X.X EOF cat \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; \u0026gt; /etc/resolv.conf nameserver 1.1.1.1 nameserver 1.0.0.1 EOF setup-hostname XXXX rc-service networking restart If the provider uses DHCP, keep the interface as DHCP instead and let Alpine request the address normally.\nRestore Without A Console Or Rescue Mode The cheapest tiers take the rest of it away too: no custom-image import, no VNC, no serial console, no rescue mode. The stock OS and SSH are all you get. The initramfs trick above is unreachable there, because its one console step — interrupting GRUB — has no SSH equivalent: an initramfs shell has no network stack and no sshd.\nWhat still works is having a script arrange that environment for you. bin456789/reinstall rewrites the stock bootloader to boot a minimal Alpine that runs entirely in RAM, and from there downloads your image and dds it over the whole disk before rebooting into it. No console is involved at any point. You watch the progress over SSH, and if the download or the write fails, the RAM system keeps its sshd up, so a failure is a retry by hand rather than a reinstall ticket.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 # From the provider\u0026#39;s stock OS, as root. This is destructive: it will # wipe the whole disk, other partitions included. # # Mainland-China boxes often cannot reach raw.githubusercontent.com; the # project mirrors the script at cnb.cool for exactly that case. curl -O https://raw.githubusercontent.com/bin456789/reinstall/main/reinstall.sh # curl -O https://cnb.cool/bin456789/reinstall/-/git/raw/main/reinstall.sh # Prompts for a username/password, used to log in over SSH and watch. # .gz / .xz / .zst compressed raw images are accepted directly. bash reinstall.sh dd --img=\u0026#34;https://example.com/alpine.img.gz\u0026#34; # Still reversible until here: `bash reinstall.sh reset` restores the # original bootloader config. reboot Before it does anything, the script asks for a username and a password, and it is worth being clear about what they are not: the new system\u0026rsquo;s login. They exist only inside the RAM installer environment — they are what you would type to SSH back in while the dd runs, to watch progress or clean up after a failure. A Linux image gets nothing injected; the script\u0026rsquo;s own end-of-run summary says as much, printing Password: [Depends on image] under \u0026ldquo;After Install\u0026rdquo;. So press Enter twice — root, then a random password the summary displays for copying — and let whatever the image baked in be the machine\u0026rsquo;s real credentials. --ssh-key replaces that password with one of your public keys for the installer environment, if typing nothing at all is the goal.\nWhen you do log in mid-install, expect a host-key mismatch: the RAM environment presents its own host keys, and so does the finished image, so the client will complain about the stock OS\u0026rsquo;s key on the way in and about both of them after the final reboot. That is the process working, not something broken — clear the old entry with ssh-keygen -R or accept the new key each time. On a NAT\u0026rsquo;d box the installer\u0026rsquo;s sshd still listens on port 22 internally, so the provider\u0026rsquo;s forwarded port keeps working throughout.\nFour things to get right, all of them consequences of there being no console:\nThe image URL must be reachable from the VPS. The download runs inside the RAM environment, over the same network the stock OS uses. For a box in mainland China that usually rules out GitHub-hosted files; one of my own servers or local object storage works better.\nThe image has to work as-is. dd mode does not modify a Linux image, so the network settings, the SSH keys, and the sshd port must be baked into the image before flashing — the \u0026ldquo;apply provider-specific network settings\u0026rdquo; step above has to happen inside the image, not on the machine afterwards. On a NAT\u0026rsquo;d box, where the provider forwards one public port to (say) internal port 22, changing the sshd port in the image orphans that forward and the machine. If the image comes from the builder, the 10-network and 20-ssh hooks are where this goes.\nCheck whether the stock OS keeps its root on LVM before rebooting. The script stages reinstall-vmlinuz and reinstall-initrd in the stock OS\u0026rsquo;s root directory and lets GRUB find them with search --file. GRUB\u0026rsquo;s LVM support is incomplete — thin-provisioned volumes are unsupported outright, and some perfectly ordinary LVM roots also defeat it for reasons that are not understood (issue #355). When that happens, GRUB stops at file '/reinstall-vmlinuz' not found and the RAM environment is never reached. On a machine with a console that is a visible error; on this kind of machine the box reboots and simply never speaks again. One lsblk settles it beforehand — if / sits on an lvm device, copy both files to /boot, which Debian-style LVM installs keep on a plain partition that GRUB reads reliably:\n1 2 lsblk -o NAME,TYPE,FSTYPE,MOUNTPOINTS # is / on an lvm device? cp /reinstall-vmlinuz /reinstall-initrd /boot/ The copies are harmless if they were not needed: search --file scans every filesystem GRUB can read, so it finds the /boot copies whenever the LVM ones are invisible. The workaround is the tool author\u0026rsquo;s own, from the issue thread. If /boot is only a directory inside the LVM root, there is no plain partition to stage onto, and I would not run this flow on such a box without a console.\nAudit before the final reboot. --hold 2 stops after the dd finishes but before the reboot. SSH back in, mount the freshly written root and boot partitions read-write, and check /etc/network/interfaces, the authorized keys, and the bootloader config while a mistake is still fixable with a text editor. It is the last moment anything is.\nThe RAM environment is small — the project lists 256 MB of RAM as the minimum — so this works on boxes too tiny for anything else. To drive the final dd by hand instead, bash reinstall.sh alpine --hold 1 boots the same RAM Alpine and stops there, nothing written; it is also a cheap dry run for whether that environment\u0026rsquo;s networking works on the provider\u0026rsquo;s NAT at all. And if the machine still does not come back after the last reboot, the floor is the provider\u0026rsquo;s own reinstall button, back to the stock OS — annoying, not fatal.\nThings I Bake Into The Image The base install above is intentionally small. The sections below are optional knobs I usually apply before making the reusable image.\nSSH For a private forwarding box, I allow root SSH and TCP forwarding. Use key auth and firewall rules if this machine is reachable from the public Internet.\nAdd or change these lines in /etc/ssh/sshd_config:\n1 2 3 4 PermitRootLogin yes AllowTcpForwarding yes ClientAliveInterval 60 ClientAliveCountMax 3 Restart SSH:\n1 rc-service sshd restart ClientAliveInterval 60 plus ClientAliveCountMax 3 makes dead SSH forwarding sessions disappear after roughly three minutes instead of waiting for long TCP timeouts.\nCommunity Repository Many useful packages live in community.\nEither run:\n1 setup-apkrepos -c Or edit /etc/apk/repositories and uncomment the matching community repository:\n1 # http://dl-cdn.alpinelinux.org/alpine/\u0026lt;alpine version\u0026gt;/community Then refresh indexes:\n1 apk update IPv6 DHCP SLAAC may work without any configuration. Some cloud providers need DHCPv6, in which case add an IPv6 stanza to /etc/network/interfaces:\n1 2 3 auto eth0 iface eth0 inet dhcp iface eth0 inet6 dhcp Install the DHCP and interface tooling if your image does not already have it:\n1 2 apk add dhcpcd ifupdown-ng rc-service networking restart Chrony If NTP was not configured during install:\n1 2 3 4 5 6 7 8 9 apk add chrony vi /etc/chrony/chrony.conf # Use `ntp.aliyun.com` if you are in mainland China; `pool.ntp.org` # is often unreliable from there. Elsewhere, the default pool is fine. # # Also add `makestep 1.0 -1` so chrony can correct large time offsets # on startup, which is common in VMs. You can remove the default # `initstepslew ...` line if it exists. The result should look like this:\n1 2 3 4 5 6 pool pool.ntp.org iburst initstepslew 10 pool.ntp.org driftfile /var/lib/chrony/chrony.drift rtcsync cmdport 0 makestep 1.0 -1 Enable it:\n1 2 rc-update add chronyd default rc-service chronyd restart Log Caps Alpine does not install a full log rotation stack by default. On a 1 GB disk, I prefer a tiny periodic script that keeps one backup and truncates logs in place so daemons can continue writing.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 cat \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; \u0026gt; /etc/periodic/hourly/logtruncate #!/bin/sh # Rotate log files larger than 4M. # Keep one .0 backup and truncate the original in place. LOG_DIR=\u0026#34;/var/log\u0026#34; MAX_SIZE=$((4 * 1024 * 1024)) # 4M in bytes find \u0026#34;$LOG_DIR\u0026#34; -type f ! -name \u0026#39;*.0\u0026#39; | while read -r file; do # Works with both GNU and BusyBox stat. size=$(stat -c %s \u0026#34;$file\u0026#34; 2\u0026gt;/dev/null || stat -f %z \u0026#34;$file\u0026#34; 2\u0026gt;/dev/null || echo 0) if [ \u0026#34;$size\u0026#34; -gt \u0026#34;$MAX_SIZE\u0026#34; ]; then logger -t logtruncate \u0026#34;Rotating $file ($size bytes)\u0026#34; # Copy current content to .0 backup, overwriting the old backup. cp \u0026#34;$file\u0026#34; \u0026#34;$file.0\u0026#34; # Preserve the inode so daemons can continue writing. truncate -s 0 \u0026#34;$file\u0026#34; logger -t logtruncate \u0026#34;Rotated $file (backup: $file.0)\u0026#34; fi done EOF chmod +x /etc/periodic/hourly/logtruncate rc-update add crond default rc-service crond start Then tell BusyBox syslog not to rotate by itself:\n1 2 sed -i \u0026#39;s/^SYSLOGD_OPTS=.*/SYSLOGD_OPTS=\u0026#34;-t -s 0\u0026#34;/\u0026#39; /etc/conf.d/syslog rc-service syslog restart APK Cache Cleanup If apk caching is enabled later, clean old packages periodically:\n1 2 3 4 5 6 7 cat \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; \u0026gt; /etc/periodic/daily/apkcacheclean #!/bin/sh find /var/cache/apk -type f -name \u0026#39;*.apk\u0026#39; -mtime +7 -delete find /var/cache/apk -type f -name \u0026#39;*.tar.gz\u0026#39; -mtime +90 -delete EOF chmod +x /etc/periodic/daily/apkcacheclean Timezone 1 2 apk add tzdata setup-timezone -z Asia/Shanghai # replace with your timezone ZeroTier Alpine\u0026rsquo;s ZeroTier package can lag behind upstream, so I usually use static binaries from zerotier-static.\nAfter placing zerotier-one at /usr/local/bin/zerotier-one, create an OpenRC service:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 cat \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; \u0026gt; /etc/init.d/zerotier-one #!/sbin/openrc-run depend() { after network-online want cgroups } start_pre() { /sbin/modprobe tun } supervisor=supervise-daemon name=zerotier-one command=\u0026#34;/usr/local/bin/zerotier-one\u0026#34; command_args=\u0026#34; \\ \u0026gt;\u0026gt;/var/log/zerotier-one.log 2\u0026gt;\u0026amp;1\u0026#34; output_log=/var/log/zerotier-one.log error_log=/var/log/zerotier-one.log pidfile=\u0026#34;/var/run/zerotier-one.pid\u0026#34; respawn_delay=5 respawn_max=0 EOF chmod +x /etc/init.d/zerotier-one rc-update add zerotier-one default rc-service zerotier-one start ZRAM Swap Zram is more useful than disk swap. Install zram-init:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 apk add zram-init # Adjust as needed. The default config creates swap and /tmp on zram. # # I usually set swap zram to roughly the same size as RAM and use lz4 # for speed. Because zram compresses memory, that does not mean it # immediately consumes that much physical RAM. # # To set swap zram to the same size as RAM, remove the default # `size0=512M` line and use: # size0=`LC_ALL=C free -m | awk \u0026#39;/^Mem:/{print int($2)}\u0026#39;` # # To use lz4: # algo0=lz4 # # The /tmp zram can be left untouched. If you want a smaller /tmp, use # something like: # size1=`LC_ALL=C free -m | awk \u0026#39;/^Mem:/{print int($2/4)}\u0026#39;` # # Remove `blck1=1024` if your default config has it. The minimum block # size is 4 KiB, so 1024 bytes is not valid. # # If /tmp is already mounted as tmpfs in /etc/fstab, remove that line # when /tmp is provided by zram-init. vi /etc/conf.d/zram-init rc-update add zram-init boot rc-service zram-init start These sysctls make the kernel more willing to use compressed swap:\n1 2 3 4 5 6 7 8 cat \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; \u0026gt; /etc/sysctl.d/01-zram.conf vm.swappiness = 180 vm.watermark_boost_factor = 0 vm.watermark_scale_factor = 125 vm.page-cluster = 0 EOF sysctl -p /etc/sysctl.d/01-zram.conf Cloud-init cloud-init is convenient if the image will be imported into a cloud provider, but it pulls in Python and is not small.\n1 2 3 apk add cloud-init # Read the installation notes displayed by apk. It will show which # cloud-init services should be enabled for your target environment. Read the package message after installation and enable the OpenRC services it lists. If you want automatic partition growth on first boot, also install:\n1 apk add cloud-utils-growpart For the smallest image, skip cloud-init and inject network configuration another way.\nExpand Root After Restoring To A Larger Disk If you restore the QCOW2 to a larger disk, grow the root partition and then grow Btrfs.\nRun:\n1 fdisk /dev/vda Inside fdisk:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 p # print; write down the current start sector of vda2, usually 133120 d # delete partition 2 # select vda2 n # new partition p # primary 2 # partition number 2 133120 # use the exact old start sector \u0026lt;ENTER\u0026gt; # default end, use the rest of the disk \u0026gt; If asked: \u0026gt; Partition #2 contains a btrfs signature. \u0026gt; Do you want to remove the signature? [Y]es/[N]o: N # DO NOT remove the signature p # verify vda2 starts at the same sector and uses the rest of the disk w # write changes Ask the kernel to reread the partition table:\n1 2 apk add parted partprobe /dev/vda || reboot If you had to reboot, continue after the reboot:\n1 2 btrfs filesystem resize max / df -h / I prefer this manual method over blindly using a grow tool because it actually expands and aligns, while some grow tools may not expand the partition to be 4K aligned.\nBBR Congestion Control For forwarding boxes, I usually enable BBR:\n1 2 3 4 5 6 cat \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; \u0026gt; /etc/sysctl.d/02-bbr.conf net.core.default_qdisc = fq net.ipv4.tcp_congestion_control = bbr EOF sysctl -p /etc/sysctl.d/02-bbr.conf Verify:\n1 sysctl net.ipv4.tcp_congestion_control Podman If I need containers on a tiny VM, I prefer Podman over Docker because the entire stack (no containerd, crun instrad of runc, lighter daemon).\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 apk add podman # Keep the overlay storage driver. It is usually more stable and has # better performance for containers, even though the underlying # filesystem is Btrfs. # # Limit container logs to 1M so they do not fill the disk. sed -i \u0026#39;s/^#log_size_max =.*/log_size_max = 1048576/\u0026#39; /etc/containers/containers.conf # Enable cgroups v2 for better compatibility with modern container runtimes. rc-update add cgroups default rc-service cgroups start # Start containers that have a restart policy such as always or unless-stopped. rc-update add podman default rc-service podman start If you want Docker CLI and Compose compatibility against the Podman socket:\n1 2 3 4 apk add docker-cli docker-cli-compose docker context create podman --docker \u0026#34;host=unix:///run/podman/podman.sock\u0026#34; docker context use podman I do not use podman-docker or podman-compose here; the Docker CLI is closer to what my existing Compose files expect.\nUFW If the provider does not give you a firewall, configure one before exposing the VM:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 apk add ip6tables ufw # Start ufw later, after the allow rules are in place. # Default policies ufw default deny incoming ufw default allow outgoing # Allow loopback ufw allow in on lo # ------------------------- # SSH # ------------------------- # SSH port; change this if you use a non-standard SSH port. ufw allow 22/tcp comment \u0026#39;SSH\u0026#39; # Optional: rate limit SSH brute force. # ufw limit 22/tcp # ------------------------- # Private / intranet ranges # ------------------------- # RFC1918 IPv4 ufw allow from 10.0.0.0/8 ufw allow from 172.16.0.0/12 ufw allow from 192.168.0.0/16 # Optional CGNAT range # ufw allow from 100.64.0.0/10 # Optional IPv6 ULA ufw allow from fc00::/7 # Also allow forwarding to RFC1918 and ULA ranges if this server is a # gateway or VPN server, or if you use Podman containers with published # ports. Published container ports usually pass through DNAT and are # handled by forward rules. # # Note that if you are using Podman containers with published ports, # these route rules can allow access to all published ports even if the # input rules only allow selected ports. If that is not what you want, # only allow specific forwarded ports here. # ufw route allow to 10.0.0.0/8 # ufw route allow to 172.16.0.0/12 # ufw route allow to 192.168.0.0/16 # ufw route allow to fc00::/7 # ------------------------- # Public services # ------------------------- # Web # ufw allow 80/tcp # ufw allow 443/tcp # ufw allow 443/udp # Example app ports # ufw allow 3000/tcp # ufw allow 9090/tcp # ------------------------- # Logging # ------------------------- ufw logging low # Enable firewall ufw enable rc-update add ufw default rc-service ufw start If this machine routes VPN, overlay network, or container traffic, add explicit ufw route allow ... rules for the private ranges you actually need.\nsshguard I avoid Fail2ban on this image because Python is too heavy for the target machine. sshguard is small and good enough.\n1 apk add sshguard nftables Do not blindly start the nftables service before checking its default ruleset from a console. On one of my Alpine images, doing that installed a default deny ruleset and locked out remote SSH. I let sshguard manage its own nftables sets instead.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 # IMPORTANT: Do not run `rc-update add nftables \u0026amp;\u0026amp; rc-service nftables start` # unless you have checked the ruleset from a console. We will let sshguard # manage its nftables sets. mkdir -p /var/lib/sshguard cat \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; \u0026gt; /etc/sshguard.conf # Full path to backend executable. Required; there is no default. BACKEND=\u0026#39;/usr/libexec/sshg-fw-nft-sets\u0026#39; # Space-separated list of log files to monitor. FILES=\u0026#39;/var/log/messages\u0026#39; # Block attackers when their cumulative attack score exceeds THRESHOLD. # Most attacks have a score of 10. THRESHOLD=20 # Block attackers for initially BLOCK_TIME seconds after exceeding THRESHOLD. # Subsequent blocks increase by a factor of 1.5. BLOCK_TIME=180 # Remember potential attackers for up to DETECTION_TIME seconds before # resetting their score. DETECTION_TIME=3600 # Permanently blacklist attackers when their cumulative score exceeds threshold. BLACKLIST_FILE=100:/var/lib/sshguard/blacklist.db # Size of IPv6 subnet to block. Defaults to a single address. IPV6_SUBNET=48 # Size of IPv4 subnet to block. Defaults to a single address. IPV4_SUBNET=24 EOF rc-update add sshguard default rc-service sshguard start Common Tools This is no longer minimal, but it makes Alpine feel closer to a general-purpose server if you find busybox too limited:\n1 2 apk add bash coreutils findutils tar grep sed gawk diffutils procps util-linux shadow curl wget iproute2 bind-tools gcompat pciutils apk add netcat-openbsd socat tcpdump iftop iptraf-ng ethtool traceroute zsh git htop tmux vim less jq iperf3 sysstat rsync file And other tools\n1 apk add bat dufs gdu lsd yazi Final Notes The most important part of this setup is the partitioning discipline. On a 1 GB disk, a lazy 512 MB /boot, a swap partition, or unbounded logs will consume the entire machine faster than the application does.\nWith a 64 MB /boot, compressed Btrfs root, no disk swap, and a few periodic cleanup jobs, Alpine stays usable even on tiny cloud disks. It is not luxurious, but it is enough for a forwarding node or a small single-purpose service.\nSetting up disks manually - Alpine Linux\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nsetup-alpine - Alpine Linux Documentation\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nSystem Disk Mode - Alpine Linux\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nBtrfs - Alpine Linux\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nBootloaders - Alpine Linux\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","date":"2026-06-25T15:00:00+08:00","permalink":"https://charlie0129.github.io/blog/p/alpine-minimal-btrfs-install/","title":"Minimal Alpine Linux on a 1 GB Btrfs Root Disk"},{"content":"I recently tried to build QEMU on macOS and make the result portable enough that it could still run after uninstalling the Homebrew packages used during the build, or I can move the whole QEMU directory to another machine. I don\u0026rsquo;t want the Homebrew build because it installs too many dependencies on the system, and I want a more self-contained bundle, not a system-wide installation.\nThe short version: fully static linking on macOS is not really the right target. macOS binaries still dynamically link against Apple system libraries such as libSystem. But for tools like QEMU, what I really wanted was not a “pure static binary”; I wanted a self-contained directory like this:\n1 2 3 4 5 6 7 8 9 10 11 _qemu-11.0.1/ ├── bin/ │ ├── qemu-img │ ├── qemu-system-x86_64 │ └── ... └── lib/ ├── libglib-2.0.0.dylib ├── libgobject-2.0.0.dylib ├── libintl.8.dylib ├── libpcre2-8.0.dylib └── ... The binaries in bin/ should load their non-system dependencies from ../lib (note that it\u0026rsquo;s relative, not absolute, so it will work even if the whole directory is moved), instead of from /opt/homebrew.\nThis post records the process, using QEMU as the example.\nThe original problem I built QEMU roughly like this:\n1 2 3 4 5 6 # You should refer to https://wiki.qemu.org/Hosts/Mac for the latest build instructions. This is just an example. brew install libffi gettext glib pkg-config zstd # may not be the exact set of dependencies you need ./configure --prefix=\u0026#34;$HOME/bin/_qemu-11.0.1\u0026#34; make -j\u0026#34;$(sysctl -n hw.ncpu)\u0026#34; make install The build worked. But after uninstalling some Homebrew dependencies, running qemu-img failed:\n1 2 3 4 dyld[82783]: Library not loaded: /opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib Referenced from: /Users/charlie/bin/_qemu-11.0.1/bin/qemu-img Reason: tried: \u0026#39;/opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib\u0026#39; (no such file) zsh: abort qemu-img This means the installed QEMU binary still contains an absolute dependency path:\n1 /opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib We can confirm that with:\n1 otool -L \u0026#34;$PREFIX/bin/qemu-img\u0026#34; Example output:\n1 2 3 4 5 qemu-img: /opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib (compatibility version 8801.0.0, current version 8801.0.0) /opt/homebrew/opt/zstd/lib/libzstd.1.dylib (compatibility version 1.0.0, current version 1.5.7) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0) ... The /usr/lib and /System/Library entries are fine. They are macOS system libraries. The /opt/homebrew/... entries are the ones that make the binary depend on my local Homebrew installation.\nWhy not just build QEMU fully static? On Linux, the natural thought is “just build a static binary”.\nOn macOS, that is not usually how things work. macOS does not support the same style of fully static userland binary that Linux users often expect. You can try to statically link some third-party libraries, but the final program will still dynamically link to Apple system libraries.\nSo the practical goal is:\nBundle third-party .dylib dependencies next to the program, and rewrite Mach-O load commands so the binary loads those local copies.\nThis is similar in spirit to what many .app bundles do, but here I wanted a plain CLI directory layout.\nThe tools involved macOS Mach-O binaries can be inspected and patched with these tools:\n1 2 3 4 otool -L \u0026lt;file\u0026gt; install_name_tool -change \u0026lt;old\u0026gt; \u0026lt;new\u0026gt; \u0026lt;file\u0026gt; install_name_tool -id \u0026lt;new-id\u0026gt; \u0026lt;dylib\u0026gt; install_name_tool -add_rpath \u0026lt;rpath\u0026gt; \u0026lt;file\u0026gt; otool -L shows dynamic library load commands.\nFor example:\n1 otool -L \u0026#34;$PREFIX/bin/qemu-img\u0026#34; might show:\n1 2 3 4 5 qemu-img: /opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib (compatibility version 8801.0.0, current version 8801.0.0) /opt/homebrew/opt/zstd/lib/libzstd.1.dylib (compatibility version 1.0.0, current version 1.5.7) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0) ... Those can be rewritten with:\n1 2 3 4 install_name_tool -change \\ /opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib \\ \u0026#39;@executable_path/../lib/libglib-2.0.0.dylib\u0026#39; \\ \u0026#34;$PREFIX/bin/qemu-img\u0026#34; @executable_path means “the directory containing the executable being launched”.\nSo it will try to find libglib-2.0.0.dylib in ../lib/ relative to the executable, no longer looking in /opt/homebrew.\nFirst attempt: patch direct dependencies only My first attempt was simple:\n1 2 3 4 5 otool -L \u0026#34;$PREFIX\u0026#34;/bin/* \\ | grep /opt/homebrew \\ | sort \\ | uniq \\ | awk \u0026#39;{print $1}\u0026#39; This found direct Homebrew dependencies used by QEMU binaries, such as:\n1 2 3 4 5 /opt/homebrew/opt/glib/lib/libgio-2.0.0.dylib /opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib /opt/homebrew/opt/glib/lib/libgmodule-2.0.0.dylib /opt/homebrew/opt/glib/lib/libgobject-2.0.0.dylib /opt/homebrew/opt/zstd/lib/libzstd.1.dylib Then I copied those libraries into $PREFIX/lib and patched the binaries.\nThis fixed some errors, but not all of them.\nRunning qemu-img then failed with a different error:\n1 2 dyld[97126]: Library not loaded: /opt/homebrew/opt/gettext/lib/libintl.8.dylib Referenced from: /Users/charlie/bin/_qemu-11.0.1/lib/libglib-2.0.0.dylib So I realized that it is not enough to patch only the binaries. The copied .dylib files have their own dependencies too.\nIn this case:\n1 2 3 qemu-img -\u0026gt; libglib-2.0.0.dylib -\u0026gt; libintl.8.dylib libintl.8.dylib was not directly referenced by qemu-img. It was a transitive dependency of libglib.\nThe correct model The dependency graph looks like this:\n1 2 3 4 5 6 7 bin/qemu-img -\u0026gt; /opt/homebrew/opt/glib/lib/libglib-2.0.0.dylib -\u0026gt; /opt/homebrew/opt/zstd/lib/libzstd.1.dylib lib/libglib-2.0.0.dylib -\u0026gt; /opt/homebrew/opt/gettext/lib/libintl.8.dylib -\u0026gt; /opt/homebrew/opt/pcre2/lib/libpcre2-8.0.dylib So the bundling process must be recursive:\nScan Mach-O files in bin/. Find its /opt/homebrew dependencies. Copy those .dylib files into lib/. Patch the binaries to refer to the bundled copies (../lib). Scan the copied .dylib files. Copy and patch their dependencies too. Repeat until no /opt/homebrew dependencies remain. Which paths should be used? For executables in bin/, I use:\n1 @executable_path/../lib/libfoo.dylib For dylibs inside lib/, I use:\n1 @loader_path/libfoo.dylib @executable_path is relative to the main executable being launched.\n@loader_path is relative to the Mach-O file that is doing the loading. For dependencies between dylibs, this is what we want:\n1 2 lib/libglib-2.0.0.dylib -\u0026gt; @loader_path/libintl.8.dylib Since both files are in the same lib/ directory, this resolves correctly.\nThe final bundling script I ended up writing a small Python script called macho-bundle-deps. You can find it here: https://github.com/charlie0129/dotfiles/blob/f15791054e517f6b5afc892312f1db73f331d475/bin/darwin/macho-bundle-deps. This is a permanent link to a specific commit, so you may want to check the latest version in the repository.\nUsage:\n1 2 3 4 5 6 PREFIX=\u0026#34;$HOME/bin/_qemu-11.0.1\u0026#34; macho-bundle-deps \\ --lib-dir \u0026#34;$PREFIX/lib\u0026#34; \\ --prefix /opt/homebrew \\ \u0026#34;$PREFIX\u0026#34;/bin Or, from inside the QEMU prefix:\n1 2 3 cd \u0026#34;$HOME/bin/_qemu-11.0.1\u0026#34; macho-bundle-deps --lib-dir lib bin The script does the following:\nfinds Mach-O files from the input paths scans their dependencies with otool -L copies matching dependencies into --lib-dir patches executable dependencies to @executable_path/../lib/... patches bundled dylib dependencies to @loader_path/... patches copied dylib IDs with install_name_tool -id recursively processes newly copied dylibs reports an error if matching absolute dependency paths remain Verifying the result After running the bundler, I verify that no Homebrew paths remain:\n1 otool -L \u0026#34;$PREFIX\u0026#34;/bin/* \u0026#34;$PREFIX\u0026#34;/lib/*.dylib | grep /opt/homebrew Now I can uninstall the dependencies from Homebrew, and the bundled QEMU still works:\n1 2 brew uninstall libffi gettext glib qemu-img --help Notes and limitations This is not the same as producing a fully static binary.\nThe result still depends on macOS system libraries, which is normal:\n1 2 /usr/lib/libSystem.B.dylib /System/Library/Frameworks/... It also does not magically make a binary portable across all macOS versions or CPU architectures. A binary built on Apple Silicon is still an arm64 Mach-O binary unless built otherwise.\nThis approach is mainly useful for making a local CLI tool directory self-contained with respect to Homebrew dependencies.\nWhy copying symlinks should be avoided Homebrew often has dylib symlinks like:\n1 libintl.dylib -\u0026gt; libintl.8.dylib When bundling, prefer copying the symlink target, not the symlink itself.\nIn shell, that means:\n1 cp -L source.dylib \u0026#34;$PREFIX/lib/\u0026#34; In Python, shutil.copy2(..., follow_symlinks=True) does the same thing.\nThis avoids ending up with a bundled symlink pointing to a non-existent file.\nFinal layout After bundling, the QEMU directory looks like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 _qemu-11.0.1/ ├── bin/ │ ├── qemu-img │ ├── qemu-io │ ├── qemu-nbd │ ├── qemu-system-aarch64 │ ├── qemu-system-x86_64 │ └── ... └── lib/ ├── libffi.8.dylib ├── libgio-2.0.0.dylib ├── libglib-2.0.0.dylib ├── libgmodule-2.0.0.dylib ├── libgobject-2.0.0.dylib ├── libintl.8.dylib ├── libpcre2-8.0.dylib └── libzstd.1.dylib Now qemu-img no longer cares whether Homebrew’s glib, gettext, pcre2, or zstd packages are installed.\nConclusion The main lesson is that macOS dependency bundling is graph traversal, not a one-shot patch.\nPatching only the top-level binaries is not enough. You also need to patch the dylibs that you copied, and then patch the dylibs that those dylibs depend on.\nThe final strategy is:\n1 scan -\u0026gt; copy -\u0026gt; patch -\u0026gt; scan copied dylibs -\u0026gt; repeat For executables:\n1 2 /opt/homebrew/.../libfoo.dylib -\u0026gt; @executable_path/../lib/libfoo.dylib For bundled dylibs:\n1 2 /opt/homebrew/.../libbar.dylib -\u0026gt; @loader_path/libbar.dylib And for the dylib’s own install name:\n1 2 /opt/homebrew/.../libfoo.dylib -\u0026gt; @loader_path/libfoo.dylib ","date":"2026-06-08T12:25:00+08:00","permalink":"https://charlie0129.github.io/blog/p/bundling-macos-dynamic-libraries/","title":"Bundling macOS Dynamic Libraries"},{"content":"背景 最近老家重建，家里需要重新布置无线网络。考虑到自建别墅面积比较大，单纯使用单个无线路由器覆盖不够，因此多个 AP 肯定是需要的。家里在装修的时候就每个房间吊顶中有留超六类线缆，这种情况对 AP 来说是非常友好的，因此决定使用企业级无线接入点（AP）来组建无线网络。由于华为系列的 AP 在国内使用量较大，闲鱼上非常好捡公司下架下来的二手设备，因此考虑华为系列的 AP。这些企业级 AP 并不需要担心二手问题，他们的质量是消费级产品无法比拟的，稳定性也是非常强。\n目前是 2025 年 12 月，虽然在国内 Wi-Fi 7 （802.11be）的 6 GHz 频段还没批准无法使用（损失不少性能），但是上一下 Wi-Fi 7 追一下新也没什么问题。华为 Wi-Fi 7 AP 是 AirEngine xx7x 产品，我闲鱼花了 500 多，激情下单了一台 AirEngine 5773-21 （有一个 2.5 Gbe PoE 口）。先买一台测测，为之后全屋部署积累经验，如果好用，再给每个房间买一台（后续更新）。\n回来一查，这设备保修到 2028 年 3 月，现在才 2025 年 12 月，还有快 3 年的保修期，很新啊。\n我目前的打算是不购买单独的 AC 控制器（AC 用于统一管理其他 AP ），因为我没有几十上百个 AP 需要管理。对于只需管理十几台 AP 来说，AC 控制器的成本和复杂度都不划算。因此打算使用华为 AP 自带的 Leader AP 模式组网（按道理华为的 AirEngine 系列大部分都是支持的，但是对于我买的 AirEngine 5773-21 来说，这里是个大坑，后面会讲）。这种模式下，Leader AP 可以担任 AC 的角色，统一管理其他 FIT AP ，每个 Leader AP 具体能管理几台 FIT AP 可以上华为 Info-Finder 上查询（基本上都大于 16 台）。\n（组网方案后续更新，包括多 AP 的组网方案， VLAN 配置 IOT 设备隔离等）\n首次启动 插上网线（注意，如果使用 PoE 的话需要满足 802.3at ，否则可能降速，或者使用 DC 供电）\n注意，开机过程中状态灯常亮，等到状态灯快速闪烁说明开机完成等待配置。\n现在华为的 AP 默认会用 DHCP 获取 IP 地址，因此只需根据 AP 的 MAC 地址（在 AP 背后会写）在 DHCP 服务器上找到对应的 IP 地址，就可以知道 AP 目前的 IP 地址了\n然后浏览器打开该 IP 即可登录 Web 管理界面（并不）。\n踩坑 然后浏览器打开该 IP 即可登录 Web 管理界面（吗？？？？），我发现并不能。\n按道理华为 Wi-Fi 6 系列（比如 AirEngine 5761 ）都是可以直接登录 Web 管理页面的，Wi-Fi 7 更新不应该不行，而且官方 AirEngine 5773-21 的彩页中也说可以用做 FAT AP / Leader AP ，怎么回事？\n我后面用 SSH 登录设备（后面会说怎么登录），没有说当前是 FIT 还是 FAT 模式（老的 AP 会在 SSH 登录的时候就提示）。而且这个命令行看的我一脸懵，老的 Wi-Fi 6 AP 都是直接 system-view 就能配置，然后命令也是传统的那一套。这新的 AP 一上来就是个 MDCLI\u0026gt; 提示符（一看就是新搞的东西，里面的命令也完全变了，我根本没时间去学习这些新命令。（后面才知道，必须升级系统后，用 switch cli 从 MC-CLI 切换到传统 CLI 模式，才能用老的命令行方式配置，但是建议还是用新的 MD-CLI ，因为传统 CLI 里面基本没啥功能了）\n看了一圈文档，说可以输入 edit-config 修改配置，然后发现提示没有权限。WTF？我管理员还能没权限？后面才知道，原来没有权限其实代表当前 AP 在 FIT 模式下（这报错给用户带来多少困扰），必须切换到 FAT 模式才有权限修改配置。那么问题来了，我都没权限做任何配置，我怎么切换到 FAT 模式（况且我也没找到切换 FAT 模式的命令）？\n后面折腾一圈才发现（各种根据文档中的蛛丝马迹去猜），是我当前固件版本太老了（V600R023C10），华为并没有在早期固件中实现 FAT AP / Leader AP 功能（设备都发布了，功能还没做好是吧），必须升级到 V600R024C10 及以上版本才有该功能，写文章时最新版本 是 V600R025C00 。OK，问题找到，接下来就是升级固件了。\n但是华为是出了名的不给资料，比如我想下载新版本固件，就算我注册了账号、也绑定了 AP 的序列号、也给了华为公司名称 + 设备序列号 也审核 通过 了 AirEngine 5773-21 的资料和软件下载权限，但是你还是下不到 AirEngine 5773-21 的固件，华为官网上根本 没有 提供任何一个版本固件的下载链接（截至 2025 年 12 月），你只能下到补丁（这还是从其他 AP 找的补丁，刚好能用到 AirEngine 5773-21 上）。\n最后我只能在闲鱼花钱上找有权限的人代下的固件，版本号 V600R024C10 ，也就是第一版支持 FAT AP / Leader AP 功能的固件。本来我想下载当前最新的 V600R025C00 ，但是我没找到人能下载这个版本，遂放弃， V600R024C10 也不是不能用。\n首次设备设置 可以按住设备旁的 Default 按钮数十秒来重置设备。观察指示灯，变成常亮表示成功。重置成功后会重启，开机过程中状态灯常亮，等到状态灯快速闪烁说明开机完成等待配置。\n你需要 SSH 上去 ssh admin@\u0026lt;AP_IP_ADDRESS\u0026gt; ，默认密码是 admin@huawei.com 。首次登录会提示你修改密码，按照提示修改后会登出，下次使用新密码再登录即可。\n查看当前版本\n1 2 3 [admin@HUAWEI] MDCLI\u0026gt; display system/system-info/software-name \u0026#34;AirEngineX773_V600R023C10SPC200.cc\u0026#34; # 当前版本 我目前的版本是 V600R023C10SPC200 ，也就是 V600R023C10 （后面的 SPC200 是补丁），需要升级到 V600R024C10 及以上版本才有 FAT AP / Leader AP 功能。\n固件升级 首先需要找台机器开启 FTP 服务端，设置好用户名密码，打开读写权限（macOS App Store 中有个 QuickFTP 还是很好用的），到时候 AP 需要通过 FTP 上传和下载固件。\n然后你需要有一个 V600R024C10 及以上版本的固件文件，放到 FTP 的根目录下面。\n需要使用的固件名称通常长这样，两者都行：\nAirEngineX773_V600R024C10.cc （不带补丁的基础固件） AirEngineX773_V600R024C10SPC100.cc （带补丁的完整固件） 其中：\nAirEngineX773：设备型号，比如 AirEngine 5773 V600R024C10：版本号，R 或者 C 之后的数字越大，版本越新。 SPC100：补丁号，可选。注意，不要下成 SPH 的热补丁了，补丁通常长这样 AirEngineX773_V600R024C10SPH150.pat（有 SPH 字样，且结尾是 .pat ），需要完整的固件（.cc 结尾）。 备份当前固件 SSH 登录设备操作\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 [admin@HUAWEI] MDCLI\u0026gt; ftpc-transfer-file [(x)admin@HUAWEI]/ftpc-transfer-file MDCLI\u0026gt; command-type put [*(x)admin@HUAWEI]/ftpc-transfer-file MDCLI\u0026gt; local-file-name AirEngineX773_V600R023C10SPC200.cc # 文件名与当前 AP 版本一致，查询：display system/system-info/software-name [*(x)admin@HUAWEI]/ftpc-transfer-file MDCLI\u0026gt; remote-file-name AirEngineX773_V600R023C10SPC200.cc # 与上面保持一致即可 [*(x)admin@HUAWEI]/ftpc-transfer-file MDCLI\u0026gt; server-ipv4-address 192.168.213.54 # FTP 服务器 IP 地址 [*(x)admin@HUAWEI]/ftpc-transfer-file MDCLI\u0026gt; server-port 2121 # FTP 服务器端口 [*(x)admin@HUAWEI]/ftpc-transfer-file MDCLI\u0026gt; user-name xxxx # FTP 用户名 [*(x)admin@HUAWEI]/ftpc-transfer-file MDCLI\u0026gt; password # FTP 密码 Enter password: Confirm password: [*(x)admin@HUAWEI]/ftpc-transfer-file MDCLI\u0026gt; emit # 提交上传任务 { \u0026#34;huawei-ftpc:transfer-id\u0026#34;: 3 # 可以用这个 ID 查看任务状态 } [admin@HUAWEI] MDCLI\u0026gt; display ftpc/transfer-tasks # 检查是否备份成功 { \u0026#34;transfer-task\u0026#34;: [ { \u0026#34;transfer-id\u0026#34;: 3, \u0026#34;command-type\u0026#34;: \u0026#34;put\u0026#34;, \u0026#34;server-address\u0026#34;: \u0026#34;192.168.213.54\u0026#34;, \u0026#34;server-port\u0026#34;: 2121, \u0026#34;local-file-name\u0026#34;: \u0026#34;AirEngineX773_V600R023C10SPC200.cc\u0026#34;, \u0026#34;remote-file-name\u0026#34;: \u0026#34;AirEngineX773_V600R023C10SPC200.cc\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;succeeded\u0026#34;, # 表示成功 \u0026#34;percentage\u0026#34;: 100 } ] } 你应该可以在 FTP 服务器上看到备份下来的固件文件了。\n下载新固件 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 [admin@HUAWEI] MDCLI\u0026gt; download-upgrade-package [(x)admin@HUAWEI]/download-upgrade-package MDCLI\u0026gt; base-software-directory AirEngineX773_V600R024C10SPC100.cc # 新固件文件名，注意要和 FTP 服务器上一致 [*(x)admin@HUAWEI]/download-upgrade-package MDCLI\u0026gt; server-ip 192.168.213.54 # FTP 服务器 IP 地址 [*(x)admin@HUAWEI]/download-upgrade-package MDCLI\u0026gt; server-port 2121 # FTP 服务器端口 [*(x)admin@HUAWEI]/download-upgrade-package MDCLI\u0026gt; transfer-protocol ftp # 传输协议 [*(x)admin@HUAWEI]/download-upgrade-package MDCLI\u0026gt; user-name xxxx # FTP 用户名 [*(x)admin@HUAWEI]/download-upgrade-package MDCLI\u0026gt; password # FTP 密码 Enter password: Confirm password: [*(x)admin@HUAWEI]/download-upgrade-package MDCLI\u0026gt; emit # 提交下载任务 [*(x)admin@HUAWEI]/download-upgrade-package MDCLI\u0026gt; display software/download-result/ # 查看下载状态 { \u0026#34;file-name\u0026#34;: \u0026#34;AirEngineX773_V600R024C10SPC100.cc\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;succeeded\u0026#34;, # 表示下载成功 \u0026#34;percentage\u0026#34;: 100 } [admin@HUAWEI] MDCLI\u0026gt; display file-operation/ # 查看文件列表，确认新固件在设备上 { \u0026#34;dir\u0026#34;: [ ... { \u0026#34;file-name\u0026#34;: \u0026#34;AirEngineX773_V600R024C10SPC100.cc\u0026#34;, # 新固件文件 \u0026#34;dir-name\u0026#34;: \u0026#34;backup:/\u0026#34;, \u0026#34;attribute\u0026#34;: \u0026#34;-rw-\u0026#34;, \u0026#34;modify-time\u0026#34;: \u0026#34;2024-11-12T12:05:04Z\u0026#34;, \u0026#34;size\u0026#34;: 52394852 } ... ] } 设置新固件为启动固件 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 [admin@HUAWEI] MDCLI\u0026gt; startup-by-mode name AirEngineX773_V600R024C10SPC100.cc # 设置启动固件，注意文件名要和上面一致 [admin@HUAWEI] MDCLI\u0026gt; display cfg/startup-infos { \u0026#34;startup-info\u0026#34;: [ { \u0026#34;position\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;configed-system-software\u0026#34;: \u0026#34;AirEngineX773_V600R023C10SPC200.cc\u0026#34;, \u0026#34;current-system-software\u0026#34;: \u0026#34;AirEngineX773_V600R023C10SPC200.cc\u0026#34;, \u0026#34;next-system-software\u0026#34;: \u0026#34;AirEngineX773_V600R024C10SPC100.cc\u0026#34;, # 确认新固件已设置为下次启动固件 \u0026#34;current-cfg-file\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;next-cfg-file\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;current-patch-file\u0026#34;: \u0026#34;NULL\u0026#34;, \u0026#34;next-patch-file\u0026#34;: \u0026#34;NULL\u0026#34; } ] } 注意，要是你之前安装了 patch ，可能会导致新系统安装不上，例如\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 [admin@HUAWEI] MDCLI\u0026gt; display cfg/startup-infos { \u0026#34;startup-info\u0026#34;: [ { \u0026#34;position\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;configed-system-software\u0026#34;: \u0026#34;AirEngineX773_V600R023C10SPC200.cc\u0026#34;, # 是的，华为拼错单词了 \u0026#34;current-system-software\u0026#34;: \u0026#34;AirEngineX773_V600R023C10SPC200.cc\u0026#34;, \u0026#34;next-system-software\u0026#34;: \u0026#34;AirEngineX773_V600R023C10SPC200.cc\u0026#34;, # 发现还是老的版本 \u0026#34;current-cfg-file\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;next-cfg-file\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;current-patch-file\u0026#34;: \u0026#34;AirEngineX773_V600R023SPH151.pat\u0026#34;, # 已安装 patch \u0026#34;next-patch-file\u0026#34;: \u0026#34;AirEngineX773_V600R023SPH151.pat\u0026#34; } ] } 这时候你需要把 patch 给清掉：\n1 2 3 [admin@HUAWEI] MDCLI\u0026gt; delete-patch delete-type all ...... 然后再设置启动固件即可。\n重启设备 1 2 3 4 [admin@HUAWEI] MDCLI\u0026gt; reboot Warning: This operation will reboot the device. Are you sure you want to continue? [Y(yes)/N(no)]:y 等设备起来，浏览器访问设备 IP 地址，应该就可以看到 Web 管理界面了。\n选择右上角 FIT 按钮，改成 FAT 模式即可使用 Leader AP 功能，重启后即可变成 Leader AP ，可以管理其他 FIT AP 了。\n安装补丁（Patch） 补丁是在基础版本上修复 bug 或者增加小功能的。补丁一般靠自己注册完设备就能在官网下到，下载补丁的时候，注意产品型号不需要选择 AirEngine 5773-21 ，因为太新了，华为官网上甚至没有这个型号。直接选择全部，只要版本对的上即可，比如我当前基础版本是 V600R024C10 ，那么补丁就需要选择 V600R024C10SPH181 这种补丁，表示在 V600R024C10 基础版本上打的热补丁 SPH181 ，数字越大表示补丁越新。补丁文件名通常长这样 AirEngineX773_V600R024C10SPH181.pat ，注意是 .pat 结尾。\n下载后，放到 FTP 服务器根目录下，然后 SSH 登录设备，执行以下命令安装补丁：\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 [admin@HUAWEI] MDCLI\u0026gt; ftpc-transfer-file [(x)admin@HUAWEI]/ftpc-transfer-file MDCLI\u0026gt; command-type get [*(x)admin@HUAWEI]/ftpc-transfer-file MDCLI\u0026gt; local-file-name AirEngineX773_V600R024C10SPH181.pat # 补丁文件名，注意和 FTP 服务器上一致 [*(x)admin@HUAWEI]/ftpc-transfer-file MDCLI\u0026gt; remote-file-name AirEngineX773_V600R024C10SPH181.pat # 与上面保持一致即可 [*(x)admin@HUAWEI]/ftpc-transfer-file MDCLI\u0026gt; server-ipv4-address 192.168.213.54 # FTP 服务器 IP 地址 [*(x)admin@HUAWEI]/ftpc-transfer-file MDCLI\u0026gt; server-port 2121 # FTP 服务器端口 [*(x)admin@HUAWEI]/ftpc-transfer-file MDCLI\u0026gt; user-name xxx # FTP 用户名 [*(x)admin@HUAWEI]/ftpc-transfer-file MDCLI\u0026gt; password # FTP 密码 Enter password: Confirm password: [*(x)admin@HUAWEI]/ftpc-transfer-file MDCLI\u0026gt; emit # 提交下载任务 { \u0026#34;huawei-ftpc:transfer-id\u0026#34;: 1 } [admin@HUAWEI] MDCLI\u0026gt; display ftpc/transfer-tasks/ # 查看任务状态 { \u0026#34;transfer-task\u0026#34;: [ { \u0026#34;transfer-id\u0026#34;: 1, \u0026#34;command-type\u0026#34;: \u0026#34;get\u0026#34;, \u0026#34;server-address\u0026#34;: \u0026#34;192.168.213.54\u0026#34;, \u0026#34;server-port\u0026#34;: 2121, \u0026#34;local-file-name\u0026#34;: \u0026#34;AirEngineX773_V600R024C10SPH181.pat\u0026#34;, \u0026#34;remote-file-name\u0026#34;: \u0026#34;AirEngineX773_V600R024C10SPH181.pat\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;succeeded\u0026#34;, # 表示成功 \u0026#34;percentage\u0026#34;: 100 } ] } [admin@HUAWEI] MDCLI\u0026gt; load-patch name AirEngineX773_V600R024C10SPH181.pat load-type run # 安装补丁，注意文件名与需要安装的补丁一致 [admin@HUAWEI] MDCLI\u0026gt; display patch/operation-schedules # 查看安装进度 { \u0026#34;operation-schedule\u0026#34;: [ { \u0026#34;phase\u0026#34;: \u0026#34;load-patch\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;successful\u0026#34;, \u0026#34;schedule\u0026#34;: 100 # 等待进度 100% }, { \u0026#34;phase\u0026#34;: \u0026#34;delete-patch\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;not-started\u0026#34;, \u0026#34;schedule\u0026#34;: 0 }, { \u0026#34;phase\u0026#34;: \u0026#34;startup-next-patch\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;not-started\u0026#34;, \u0026#34;schedule\u0026#34;: 0 }, { \u0026#34;phase\u0026#34;: \u0026#34;reset-startup-patch\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;not-started\u0026#34;, \u0026#34;schedule\u0026#34;: 0 } ] } [admin@HUAWEI] MDCLI\u0026gt; display patch/patch-infos { \u0026#34;patch-info\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;AirEngineX773_V600R024C10SPH181.pat\u0026#34;, # 验证补丁安装成功 \u0026#34;version\u0026#34;: \u0026#34;V600R024C10SPH181\u0026#34;, \u0026#34;state\u0026#34;: \u0026#34;running\u0026#34;, \u0026#34;runtime\u0026#34;: \u0026#34;2025-12-13T21:39:48+08:00\u0026#34;, \u0026#34;path\u0026#34;: \u0026#34;/\u0026#34;, \u0026#34;operations\u0026#34;: { \u0026#34;operation\u0026#34;: [ { \u0026#34;position\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;position-type\u0026#34;: \u0026#34;MPU\u0026#34;, \u0026#34;upgrade-mode\u0026#34;: \u0026#34;reset-board\u0026#34; # 某些补丁需要重启设备才能生效 } ] } } ] } 你也可以在 Web 管理界面上查看补丁是否安装成功。\nBonus: 启用 IPv6 我发现 AP 下面的设备获取不到 IPv6 地址（我内网中 IPv6 配置都是正确的），基本上肯定是在 AP 导致的 IPv6 地址无法下发。\n后面了解华为 AP 默认不开 IPv6 报文转发。原因是：华为认为 IPv4 是主流，在 IPv4 网络中，如果存在较多 IPv6 协议报文，会影响无线网络性能，也会损耗设备的 CPU 处理能力。因此在纯 IPv4 网络中，可以通过不处理 IPv6 无线报文来提高 IPv4 网络性能。\n不过，我就是要用 IPv6 ，因此要去把 WLAN 处理 IPv6 报文的功能打开（ Web UI 没有，要命令行开），SSH 登录设备，执行以下命令：\n官方文档里面其实有这个，但是我找半天没找到，在：“参考-MD-CLI配置参考-WLAN配置-WLAN用户管理” 里面。不是，你把 IPv6 相关配置放在“用户管理”里面？这跟用户管理有啥关系？\n1 2 3 4 5 6 7 8 9 10 11 [admin@HUAWEI] MDCLI\u0026gt; edit-config [(gl)admin@HUAWEI] MDCLI\u0026gt; wlan-sta-access [(gl)admin@HUAWEI]/wlan-sta-access MDCLI\u0026gt; sta-ipv6-switch true [*(gl)admin@HUAWEI]/wlan-sta-access MDCLI\u0026gt; commit 之后你的设备应该就能正确收发 IPv6 报文了。\n结语 这告诉我们， AirEngine 5773-21 这种 Wi-Fi 7 新设备，还是太新了，所有的坑都得自己踩。与 Wi-Fi 6 的设备比如 AirEngine 5761 相比网上大把教程还是差太远了。不过也算是积累了一些经验，后续再买多台 AP 的时候就不会再踩这些坑了。\n这次单台 AP 就放北京用了，年底回老家再配置多 AP 组网方案。\n","date":"2025-12-13T16:00:00+08:00","permalink":"https://charlie0129.github.io/blog/p/huawei-airengine-5773-gotchas/","title":"(Chinese Only) 华为 AirEngine 5773-21 踩坑记录"},{"content":"To install ZeroTier, one would typically allow port 9993/UDP on their firewall. However, in certain scenarios, you may need to run a ZeroTier moon on non-standard ports due to network restrictions (e.g. behind NAT) or conflicts with other services. This guide will walk you through the steps to set up a ZeroTier moon on non-standard ports.\nSetup Relays (Moon Nodes) on Non-Standard Ports Install ZeroTier like usual:\n1 2 3 curl -s https://install.zerotier.com | sudo bash # and join a network (if your moon also acts as a client) zerotier-cli join \u0026lt;network_id\u0026gt; Setup moon:\n1 zerotier-idtool initmoon /var/lib/zerotier-one/identity.public \u0026gt;\u0026gt;/var/lib/zerotier-one/moon.json Here is the important part: edit the moon.json file to specify the desired non-standard ports. Open the file with your preferred text editor:\n1 vim /var/lib/zerotier-one/moon.json Modify the stableEndpoints section to include your public IP address along with the desired non-standard port. If you are behind a NAT, use your router\u0026rsquo;s public IP address and forward the same port from your server to your router. The format should be IP_ADDRESS/PORT.\nFor example, if you want to use port 14999, change the line to:\n1 \u0026#34;stableEndpoints\u0026#34;: [\u0026#34;xxx.xxx.xxx.xxx/14999\u0026#34;] Generate the moon configuration (this is what clients will use to connect to your moon):\n1 zerotier-idtool genmoon /var/lib/zerotier-one/moon.json You should have one file that looks like *.moon in your current dir. Move the generated moon file to the ZeroTier directory:\n1 2 mkdir -p /var/lib/zerotier-one/moons.d/ mv *.moon /var/lib/zerotier-one/moons.d/ If your moon also acts as a client, change the client configuration to use the non-standard port. Edit /var/lib/zerotier-one/local.conf:\n1 2 3 4 5 { \u0026#34;settings\u0026#34;: { \u0026#34;primaryPort\u0026#34;: 14999 } } Restart the ZeroTier service to apply the changes:\n1 systemctl restart zerotier-one Setup Clients (Leaf Nodes) On the client side, you have to change the default port as well. Yes, the client\u0026rsquo;s default port (9993) must match the moon\u0026rsquo;s port. Otherwise, they won\u0026rsquo;t be able to communicate in my tests.\nAfter you installed ZeroTier and joined the network, edit /var/lib/zerotier-one/local.conf:\n1 2 3 4 5 { \u0026#34;settings\u0026#34;: { \u0026#34;primaryPort\u0026#34;: 14999 } } Copy /var/lib/zerotier-one/moons.d/*.moon from the moon server to the client machine\u0026rsquo;s /var/lib/zerotier-one/moons.d/ directory, so the client knows about the moon:\n1 scp \u0026#34;user@moon-server:/var/lib/zerotier-one/moons.d/*.moon\u0026#34; /var/lib/zerotier-one/moons.d/ Restart the ZeroTier service on the client:\n1 systemctl restart zerotier-one Verification You can verify that the moon is functioning correctly by checking zerotier-cli command on the client to see if it can connect to the moon.\n1 zerotier-cli peers You should see an entry for your moon with the correct non-standard port, similar to the example below (xxx.xxx.xxx.xxx/14999):\n1 2 3 4 5 6 7 8 9 zerotier-cli peers 200 peers \u0026lt;ztaddr\u0026gt; \u0026lt;ver\u0026gt; \u0026lt;role\u0026gt; \u0026lt;lat\u0026gt; \u0026lt;link\u0026gt; \u0026lt;lastTX\u0026gt; \u0026lt;lastRX\u0026gt; \u0026lt;path\u0026gt; 35c192ce9b 1.15.3 LEAF 287 DIRECT 11575 11575 2001:19f0:6001:2c59:beef:3d:6767:df71/21006 3cdfac522d 1.16.0 MOON 66 DIRECT 3660 3660 xxx.xxx.xxx.xxx/14999 778cde7190 - PLANET 287 DIRECT 44090 43802 2605:9880:400:c3:254:f2bc:a1f7:19/9993 cafe04eba9 - PLANET 287 DIRECT 44090 43802 84.17.53.155/9993 cafe80ed74 - PLANET 261 DIRECT 269315 43837 2a02:6ea0:c87f::1/9993 cafefd6717 - PLANET 246 DIRECT 299345 43845 2a02:6ea0:d368::9993/9993 If your \u0026lt;lat\u0026gt; is -1, it means the client cannot reach the moon. Double-check your configuration and ensure that the specified ports are open and correctly forwarded if behind a NAT.\n","date":"2025-11-27T10:20:00+08:00","permalink":"https://charlie0129.github.io/blog/p/set-up-zerotier-moon-on-non-standard-ports/","title":"Set Up ZeroTier Moon on Non-Standard Ports"},{"content":"Introduction Most schools and universities in China have a limited number of GPU servers (often only one or two, or even none). This makes the management of GPU development machines quite different than corporation which have virtually unlimited GPU servers.\nLet\u0026rsquo;s consider a normal lab environment with 12 students, each needing a GPU server for their projects, and we have one GPU server with 8 GPUs available.\nIf we give each student a dedicated GPU server, we would need 12 servers, this is impractical and costly. If we let all the students share a single GPU server (all students have access to all resources), it would be very difficult to manage, as each student would need to install their own software and dependencies, leading to conflicts. Soon the GPU server will become a hot mess. Making each student use a different user account would not solve the problem, as they would still share the same OS and software environment, and will potentially break each other\u0026rsquo;s environment. If we make use of IOMMU and PCIe passthrough, we can assign a virtual machine with one dedicated passthrough\u0026rsquo;d GPU to each student. This approach has almost no interference between students, as each student has their own OS and software environment, even with it\u0026rsquo;s own GPU. However, this leads to a lot of wasted resources, because most of the time the GPU is idle, and each student will only have access to one GPU at a time, even if all of the GPUs are idle. If we use GPU virtualization, we can assign a virtual machine with a shared GPU to each student. This way, each student can use the GPU resources as needed, and the GPU can be shared among multiple students. This approach is more efficient and cost-effective, as it allows for better resource utilization. However, it requires a GPU that supports virtualization, such as NVIDIA\u0026rsquo;s vGPU or AMD\u0026rsquo;s MxGPU. Most consumer GPUs do not support virtualization, so this approach is not feasible for most schools and universities. This is where containerization comes in. By using containerization, we can create a shared GPU development server that runs a container for each student. Each container can have its own software environment, and all (or some of) of the GPUs can be shared among multiple containers. This approach is more efficient and cost-effective, as it allows for better resource utilization, and does not require a GPU that supports virtualization. However, it should be noted that this approach is not as isolated as virtual machines, as all containers share the same kernel and GPU resources. Therefore, it is important to ensure that the containers are properly configured, and that the students are aware of the limitations and potential issues that may arise from sharing the same GPU resources (for example, if one student runs a GPU-intensive task, it may affect the performance of other students\u0026rsquo; containers).\nWe will be using the CT (LXC Containers) in Proxmox VE to achieve this. Why not use Docker? Because Docker is meant for running applications, not for running full Linux distros. Although you can use something like sysbox to run a full Linux distro in Docker, i would still prefer to use LXC Containers, as they are built for this purpose and are already built into Proxmox VE, making it easier to manage and deploy.\nProxmox VE Installation Download the latest Proxmox VE ISO from the official website. At the time of writing, the latest version is Proxmox VE 9.0. Use whatever method you prefer to install Proxmox VE, such as using a USB drive.\nI will use IPMI of the server to install Proxmox VE, as it is the most convenient method for me. You can also use a monitor and keyboard to install Proxmox VE if you prefer.\nMount the Proxmox VE installation ISO as virtual media so we can boot into.\nNow reboot the server into the virtual media. Supermicro motherboards lets you invode the boot menu using F11. Your motherboard may have a different shortcut.\nChoose our CDROM virtual media.\nBoot into the Proxmox install menu (I prefer terminal UI over GUI ones).\nAfter you accepted the license, you will need to choose the target installtion disk. You should choose the boot SSDs on your server, not data drives. The drive in picture is 3*1.92TB SSDs in RAID 5.\nToggle Advanced options. Keep ext4, we will not be using ZFS. I know ZFS has CoW, snapshotting, compressing, checksums, and a ton of other features. But we will not use ZFS, because it will potentially cause some problems, e.g., high IO load when using RAID-Z zvols, extremely slow container start times when using ZFS storage driver in Docker, SSD write amplifications, low SSD random read/write performance, ZFS ARC not being given back to the OS as fast as needed on high memory pressure systems leading to OOM, and a ton of other issues that I previously encountered. To save me some trouble, I will use the battle-tested ext4.\nTotal size: keep as-is Swap size: 0, we don\u0026rsquo;t need it, our memory is large enought (1TiB) and we will later use ZRAM as swap Maximum root volume size: 100 (GiB), make it slightly larger so we can install things into the root volume, but not too large to occupy the data volume space. Other options can be left empty.\nOther install steps (Keyboard setup, root password) can proceed as you normally would do. Remember to set a complex root password.\nRegarding to IP addresses:\nIf you are going to use a static IP address like I do, you can just set it here and forget about it.\nIf you are going to use a dynamic (DHCP) addresses, you should keep what\u0026rsquo;s already in here (static IP). You don\u0026rsquo;t need to change anything. The valid IP address should already be automatically discovered from DHCP servers and filled in. After the installation, you can change the static IP mode to DHCP mode by running vi /etc/network/interfaces and make the following edits:\n1 2 3 4 5 6 7 8 auto vmbr0 - iface vmbr0 inet static + iface vmbr0 inet dhcp # Enable DHCP and remove static IP. - address 10.112.154.220/16 - gateway 10.112.0.1 bridge-ports eno1 bridge-stp off bridge-fd 0 After you enabled DHCP in the config, apply it by ifreload -a. Check if it takes effect by ip a show vmbr0, you should see something like valid_lft 7160sec preferred_lft 7160sec. If there is no xx sec inside it, your DHCP is malfunctioning.\n1 2 3 4 5 6 7 root@z8:/etc/network# ip a show vmbr0 4: vmbr0: \u0026lt;BROADCAST,MULTICAST,UP,LOWER_UP\u0026gt; mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 30:13:8b:6d:3a:a6 brd ff:ff:ff:ff:ff:ff inet 10.112.154.220/16 brd 10.112.255.255 scope global dynamic vmbr0 valid_lft 7160sec preferred_lft 7160sec inet6 fe80::3213:8bff:fe6d:3aa6/64 scope link valid_lft forever preferred_lft forever Proxmox VE Post-Installtion Log in as root as we will make a few modifications.\nChange APT Sources In China, we don\u0026rsquo;t really have a good global Internet connection, so we will change APT source to mirrors in China.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 # Change APT source mirrors sed -i \u0026#39;s/deb.debian.org/mirrors.ustc.edu.cn/g\u0026#39; /etc/apt/sources.list.d/debian.sources sed -i \u0026#39;s/security.debian.org/mirrors.ustc.edu.cn/g\u0026#39; /etc/apt/sources.list.d/debian.sources # Change Ceph source mirrors if [ -f /etc/apt/sources.list.d/ceph.sources ]; then CEPH_CODENAME=`ceph -v | grep ceph | awk \u0026#39;{print $(NF-1)}\u0026#39;` source /etc/os-release cat \u0026gt; /etc/apt/sources.list.d/ceph.sources \u0026lt;\u0026lt;EOF Types: deb URIs: https://mirrors.ustc.edu.cn/proxmox/debian/ceph-$CEPH_CODENAME Suites: $VERSION_CODENAME Components: no-subscription Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg EOF fi Remove enterprise repo and add no-subscription repo.\n1 2 3 4 5 6 7 8 9 10 11 # Remove enterprise sources rm /etc/apt/sources.list.d/pve-enterprise.sources # Add no-subscription sources cat \u0026gt; /etc/apt/sources.list.d/pve-no-subscription.sources \u0026lt;\u0026lt;EOF Types: deb URIs: https://mirrors.ustc.edu.cn/proxmox/debian/pve Suites: trixie Components: pve-no-subscription Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg EOF Change CT Template Sources Use mirrors in China.\n1 2 sed -i.bak \u0026#39;s|http://download.proxmox.com|https://mirrors.ustc.edu.cn/proxmox|g\u0026#39; /usr/share/perl5/PVE/APLInfo.pm systemctl restart pvedaemon Stop Cluster Services We are not using PVE clusters. Disable them. If you are using PVE clusters, you probably will not be reading this guide :P\n1 2 3 systemctl disable --now pve-ha-crm.service systemctl disable --now pve-ha-lrm.service systemctl disable --now corosync.service Install Common Tools Useful tools that will be used regularly.\n1 apt install htop sysstat vim sudo Configure Shell I like ZSH and my dotfiles, so I will use them.\n1 apt install git zsh 1 2 3 4 5 6 7 8 cd git clone --depth=1 https://github.com/charlie0129/dotfiles.git cd dotfiles ./bootstrap.sh -f chsh -s /usr/bin/zsh # Run zsh and follow the instructions zsh Configure ZRAM Swap 1 2 3 4 5 git clone --depth=1 https://github.com/foundObjects/zram-swap.git cd zram-swap ./install.sh cd .. rm -rf zram-swap The default lz4 is a good balance between speed and compression ratio. If you want to sacrifice speed for better compression, you can change the config to use zstd as the compression method.\n1 sed -i \u0026#39;s/_zram_algorithm=.*/_zram_algorithm=\u0026#34;zstd\u0026#34;/g\u0026#39; /etc/default/zram-swap Apply\n1 systemctl restart zram-swap Tune kernel parameters to make better use of ZRAM\n1 2 3 4 5 6 cat \u0026lt;\u0026lt;EOF \u0026gt; /etc/sysctl.d/zram.conf vm.swappiness = 180 vm.watermark_boost_factor = 0 vm.watermark_scale_factor = 125 vm.page-cluster = 0 EOF Apply\n1 sysctl --system Enable IPv6 SLAAC Proxmox disables IPv6 by default. To enable IPv6 SLAAC:\n1 2 3 4 5 6 7 8 cat \u0026lt;\u0026lt;EOF \u0026gt; /etc/sysctl.d/ipv6.conf net.ipv6.conf.default.accept_ra = 2 net.ipv6.conf.all.accept_ra = 2 net.ipv6.conf.default.forwarding = 1 net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.default.proxy_ndp = 1 net.ipv6.conf.all.proxy_ndp = 1 EOF Apply it\n1 2 3 sysctl --system # Note: the following command may break your internet connection. systemctl restart networking TRIM Optimizations TRIM will help prolong SSD lifespan and maintain performance. Enable it on the LVM thin pool.\n1 sed -i \u0026#39;s/.*issue_discards = .*/\\tissue_discards = 1/g\u0026#39; /etc/lvm/lvm.conf Since LXCs are just on the host, it will automatically issue discards if the underlying storage supports it, so no further action is needed. If you are using VM, be sure to use SCSI disks (VirtIO SCSI Single) with discard=on option. I\u0026rsquo;ve heard that VirtIO Block devices will not work with dicards but I haven\u0026rsquo;t tested it yet. Correct me if I\u0026rsquo;m wrong.\nLimit Journal Size 1 sed -i \u0026#39;s/.*SystemMaxUse.*/SystemMaxUse=32M/g\u0026#39; /etc/systemd/journald.conf Apply\n1 2 systemctl daemon-reload systemctl restart systemd-journald NVIDIA P2P Driver Installation Instead of regular NVIDIA drivers, we will be installing P2P-enabled drivers to force enable PCIe P2P capabilities on consumer cards (like GeForce RTX 4090). This will bring performance boost (~10%) across multiple scenarios. For details, refer to my blog post: Enabling PCIe P2P on NVIDIA RTX 4090s\nDisable IOMMU Since PCIe P2P in Linux doesn\u0026rsquo;t work so well with IOMMU-enabled systems (there are many potential issues you may run into), you may as well just disable IOMMU. Note that if you have more than 255 CPU cores, you will only have access to 255 CPUs due to APIC fallback.\nTo do disable IOMMU, you can either\nDisable Intel VT-d (AMD is on by default) in BIOS Disable intel_iommu or amd_iommu in Linux kernel parameters To disable intel_iommu or amd_iommu in Linux kernel parameters, use the following command to add intel_iommu=off and amd_iommu=off to your GRUB_CMDLINE_LINUX_DEFAULT. BTW: I also reduce the screen resolution to 1024x768 because it\u0026rsquo;s a server\n1 sed -i \u0026#39;s/GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT=\u0026#34;intel_iommu=off amd_iommu=off video=1024x768@60\u0026#34;/g\u0026#39; /etc/default/grub Reboot your system to see the effect.\n1 reboot Make sure the follow command produces NO entries. If there are output, it means IOMMU is not correctly disabled.\n1 ls /sys/class/iommu Disable ACS You may need to disable ACS to get PCIe P2P to work, refer to https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html (PCI Access Control Services)\nEnable Above 4G Decoding PCIe P2P will need to access each GPU\u0026rsquo;s memory, so the PCIe BAR should be large enough to cover GPU\u0026rsquo;s memory. You must enable Above 4G Decoding in your motherboard settings to make the PCIe BAR large enough to work. Enable it in your motherboard\u0026rsquo;s BIOS.\nInstall Official Driver Before we install the P2P kernel module, we need to install the official drivers.\nSince we are installing unofficial P2P kernel modules, there is a limited number of driver version that will work. The version of the P2P kernel modules MUST be the same as the official driver we are going to install.\nRefer to https://github.com/tinygrad/open-gpu-kernel-modules to see what versions are available. I also have my patches that works with much newer driver versions (575.57.08) (at the time of writing) https://github.com/charlie0129/open-gpu-kernel-modules .\nI will be using version 575.57.08, so we should download the official driver of the same version (575.57.08). You should download the .run file, e.g., NVIDIA-Linux-x86_64-575.57.08.run\n1 2 3 # Assume you download the installer as NVIDIA-Linux-x86_64-575.57.08.run installer=\u0026#34;NVIDIA-Linux-x86_64-575.57.08.run\u0026#34; chmod +x $installer Skip kernel modules because we will install P2P-patched version later.\n1 ./$installer --no-kernel-modules Choose whatever you want, doesn\u0026rsquo;t matter.\nYes, disable nouveau, we will be using NVIDIA drivers.\nDo not abort, just continue. Nouveau will be disabled on next boot.\nJust choose the default option for every step that comes later.\nRemove the driver installation file because we will not be using it later.\n1 rm $installer You can skip reboot now. We will reboot after we installed the kernel modules.\nInstall P2P-Enabled Kernel Modules Make sure the version of the P2P-enabled kernel modules match the version of the driver. I will use veriosn 575.57.08.\n1 git clone --depth=1 -b 575.57.08-p2p https://github.com/charlie0129/open-gpu-kernel-modules.git To build kernel modules, install build dependencies and kernel source. sudo installed because the install script uses sudo but we don\u0026rsquo;t have it now.\n1 apt install sudo build-essential proxmox-headers-$(uname -r) Build and install the kernel module\n1 ./install.sh You can safely ignore the NVIDIA-SMI failure as long as the build succeeds (you should see a DEPMOD /lib/modules/6.14.8-2-pve line) because we haven\u0026rsquo;t rebooted yet (so NVIDIA-SMI can\u0026rsquo;t be used).\nRemove the source after installing\n1 2 cd .. rm -rf open-gpu-kernel-modules Reboot your system and you should see nvidia-smi running fine.\nTo tell if P2P is enabled, we will use a simple method (performance testing will be done later):\n1 nvidia-smi -q | grep -i bar -A 3 You should see \u0026gt; 2048 MiB BAR1 Total memory, 32768 MiB in my case.\n1 2 3 4 BAR1 Memory Usage Total : 32768 MiB Used : 2 MiB Free : 32766 MiB Enable Persistence Mode Persistence mode will:\nKeep the GPU driver running so program can start faster Lower GPU power mode when it\u0026rsquo;s idle to save power. For RTX 4090s, it can drop from ~70W to ~10W. Make sure /dev/nvidia* device nodes are ready. This is useful because we are passing the GPU devices /dev/nvidia* to CTs (LXC Containers) later, it requires the device nodes to be present on start up for auto-started CTs. So you really should enable it.\nTo enable persistence mode:\n1 /usr/bin/nvidia-smi -pm 1 To make it persistent across reboots, we will use a cron job.\n1 crontab -e Add a line to run nvidia-smi on boot:\n1 @reboot /usr/bin/nvidia-smi -pm 1 \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 Why not use nvidia-persistenced systemd service? Because for whatever reason, it does not initialize the device nodes /dev/nvidia* correctly, so auto-started CTs will encounter Cuda failure 'unknown error'.\n1 2 3 4 cd /usr/share/doc/NVIDIA_GLX-1.0/samples tar jxf nvidia-persistenced-init.tar.bz2 cd nvidia-persistenced-init ./install.sh You should feel nvidia-smi runs much faster than before.\nBuild CT Templates We will build a CT Template that has everything a student will need:\nCommon build dependencies CUDA Docker \u0026hellip; Download the Linux distro you want. I chose Ubuntu 24.04, not because I like Ubuntu (I use Debian), but because most students only know Ubuntu.\nCreate an unprivileged CT just like you normally would. Just remember to give it a bit more disk space (32GB or more) because we will install CUDA later and CUDA is really large.\nInstall Docker After you created the CT, shut it down. Add additional settings to allow Docker to use overlayfs driver, otherwise Docker images will be extremely inefficient in CTs.\n1 2 3 4 5 6 7 # This command should be run on the host, not CT. # Assume 8000 is the ID of your template CT cat \u0026lt;\u0026lt;EOF \u0026gt;\u0026gt; /etc/pve/local/lxc/8000.conf lxc.apparmor.profile: unconfined lxc.cgroup.devices.allow: a lxc.cap.drop: EOF All following commands should be run in the CT unless specified otherwise.\nAdd Docker configuration. Systemd journal is used as log driver to make use of previously configured journal size limit. By writing logs to a centralized location, it\u0026rsquo;s also useful if you want to write logs to memory to reduce disk writes (e.g. log2ram).\n1 2 3 4 5 6 7 8 mkdir -p /etc/docker/ cat \u0026lt;\u0026lt;EOF \u0026gt; /etc/docker/daemon.json { \u0026#34;live-restore\u0026#34;: true, \u0026#34;experimental\u0026#34;: true, \u0026#34;log-driver\u0026#34;: \u0026#34;journald\u0026#34; } EOF Install Docker\n1 2 export DOWNLOAD_URL=https://mirrors.ustc.edu.cn/docker-ce curl -fsSL https://get.docker.io | sh Revert Containerd Config Docker modified containerd config. We will revert it to the default config.\n1 containerd config default \u0026gt;/etc/containerd/config.toml Add GPUs to CT Choose which GPU you need to passthrough by looking for the index in nvidia-smi, or by ls -l /dev/nvidia*\n1 2 3 4 5 6 7 8 9 10 11 # This command should be run on the host, not CT. # ls -l /dev/nvidia* crw-rw-rw- 1 root root 195, 0 2025-08-17 01:47:58 /dev/nvidia0 crw-rw-rw- 1 root root 195, 1 2025-08-17 01:48:00 /dev/nvidia1 crw-rw-rw- 1 root root 195, 2 2025-08-17 01:48:01 /dev/nvidia2 crw-rw-rw- 1 root root 195, 3 2025-08-17 01:48:02 /dev/nvidia3 crw-rw-rw- 1 root root 195, 4 2025-08-17 01:48:03 /dev/nvidia4 crw-rw-rw- 1 root root 195, 5 2025-08-17 01:48:05 /dev/nvidia5 crw-rw-rw- 1 root root 195, 6 2025-08-17 01:48:06 /dev/nvidia6 crw-rw-rw- 1 root root 195, 7 2025-08-17 01:48:07 /dev/nvidia7 ... (omitted) For example, if want to add all 8 GPUs (0, 1, 2, 3, 4, 5, 6, 7) to CT, I will need to run:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 # This command should be run on the host, not CT. # Assume 8000 is the ID of your template CT cat \u0026lt;\u0026lt;EOF \u0026gt;\u0026gt; /etc/pve/local/lxc/8000.conf lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file lxc.mount.entry: /dev/nvidia1 dev/nvidia1 none bind,optional,create=file lxc.mount.entry: /dev/nvidia2 dev/nvidia2 none bind,optional,create=file lxc.mount.entry: /dev/nvidia3 dev/nvidia3 none bind,optional,create=file lxc.mount.entry: /dev/nvidia4 dev/nvidia4 none bind,optional,create=file lxc.mount.entry: /dev/nvidia5 dev/nvidia5 none bind,optional,create=file lxc.mount.entry: /dev/nvidia6 dev/nvidia6 none bind,optional,create=file lxc.mount.entry: /dev/nvidia7 dev/nvidia7 none bind,optional,create=file lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir lxc.mount.entry: /dev/fb0 dev/fb0 none bind,optional,create=file EOF Note the lxc.mount.entry: /dev/nvidiaX dev/nvidiaX none bind,optional,create=file lines. Each line represents a single GPU to add to the CT. I will add 8 GPUs, so I add 8 lines from nvidia0 through nvidia7. Remove or add them as needed.\nCheck if the GPUs are there inside CT:\n1 2 3 4 5 6 7 8 9 10 11 12 13 # ls -l /dev/nvidia* crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:00 /dev/nvidia-modeset crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:50:43 /dev/nvidia-uvm crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:50:43 /dev/nvidia-uvm-tools crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:47:58 /dev/nvidia0 crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:00 /dev/nvidia1 crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:01 /dev/nvidia2 crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:02 /dev/nvidia3 crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:03 /dev/nvidia4 crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:05 /dev/nvidia5 crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:06 /dev/nvidia6 crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:48:07 /dev/nvidia7 crw-rw-rw- nobody nogroup 0 B 2025-08-17 01:47:58 /dev/nvidiactl We haven\u0026rsquo;t installed drivers yet, so nvidia-smi is not available.\nInstall GPU Driver Important: you should use the same driver version as the host (575.57.08 in my case).\n1 2 3 4 wget \u0026lt;driver-runfile-url\u0026gt; -O driver.run chmod +x driver.run driver.run --no-kernel-modules rm driver.run Just install as normal.\nNow you should be able to use nvidia-smi and see the GPUs you added to the CT (8 GPUs in my case).\nInstall CUDA Important: current driver version MUST be greater than (or equal to) the driver required by CUDA. For example, CUDA 12.9.0 will work on my machine because CUDA 12.9.0 requires 575.00, and my current driver version is 575.57.08 (greater than 575.00).\n1 2 3 4 wget \u0026lt;cuda-runfile-url\u0026gt; -O cuda.run chmod +x cuda.run cuda.run rm cuda.run Remember to deselect the driver (we already installed the correct driver):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 ┌──────────────────────────────────────────────────────────────────────────────┐ │ CUDA Installer │ │ - [ ] Driver \u0026lt;== deselect this │ │ [ ] 575.51.03 \u0026lt;== deselect this │ │ + [X] CUDA Toolkit 12.9 │ │ [ ] CUDA Demo Suite 12.9 \u0026lt;== deselect this (not useful, takes up space) │ │ [ ] CUDA Documentation 12.9 \u0026lt;== deselect this (not useful, takes up space) │ │ - [ ] Kernel Objects │ │ [ ] nvidia-fs │ │ Options │ │ Install │ │ │ │ Up/Down: Move | Left/Right: Expand | \u0026#39;Enter\u0026#39;: Select | \u0026#39;A\u0026#39;: Advanced options │ └──────────────────────────────────────────────────────────────────────────────┘ After CUDA installation, configure PATH and LD_LIBRARY_PATH per printed instructions.\n1 2 3 4 5 ... (omitted) Please make sure that - PATH includes /usr/local/cuda-12.9/bin - LD_LIBRARY_PATH includes /usr/local/cuda-12.9/lib64, or, add /usr/local/cuda-12.9/lib64 to /etc/ld.so.conf and run ldconfig as root ... (omitted) Note that you can use /usr/local/cuda instead of /usr/local/cuda-XX.X so it\u0026rsquo;s independent of CUDA versions.\nAlso add export CUDA_HOME=/usr/local/cuda to your shell rc file.\nnvcc should be available after a shell reload:\n1 2 3 4 5 6 # nvcc -V nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2025 NVIDIA Corporation Built on Wed_Apr__9_19:24:57_PDT_2025 Cuda compilation tools, release 12.9, V12.9.41 Build cuda_12.9.r12.9/compiler.35813241_0 Install NCCL You will need NCCL for most GPU-related communications.\nYou should find a NCCL that\u0026rsquo;s compatible with your CUDA version. For example, NCCL 2.27.3 is compatible with my CUDA version 12.9.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 # Download NCCL package wget https://xxx/nccl_2.27.3-1+cuda12.9_x86_64.txz tar Jxf nccl_2.27.3-1+cuda12.9_x86_64.txz # Set some variables NCCL_VERSION=2.27 mkdir -p /usr/local/nccl-$NCCL_VERSION cp -vRf nccl_2.27.3-1+cuda12.9_x86_64/* /usr/local/nccl-$NCCL_VERSION rm -rf nccl_2.27.3-1+cuda12.9_x86_64 ln -s /usr/local/nccl-$NCCL_VERSION /usr/local/nccl echo \u0026#34;/usr/local/nccl/lib\u0026#34; \u0026gt;\u0026gt;/etc/ld.so.conf ldconfig ln -s /usr/local/nccl/include/nccl.h /usr/local/include/nccl.h # Remove NCCL package rm https://xxx/nccl_2.27.3-1+cuda12.9_x86_64.txz Also add export NCCL_HOME=/usr/local/nccl in your shell rc file,\nInstall NVIDIA Container Toolkit To run Docker containers with GPUs.\nNote that the URLs below are mirrors in China. You can use original URL if you prefer.\n1 2 3 4 5 6 7 8 curl -fsSL https://mirrors.ustc.edu.cn/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg curl -s -L https://mirrors.ustc.edu.cn/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \\ sed \u0026#39;s#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g\u0026#39; | \\ tee /etc/apt/sources.list.d/nvidia-container-toolkit.list sed -i \u0026#39;s/nvidia.github.io/mirrors.ustc.edu.cn/g\u0026#39; /etc/apt/sources.list.d/nvidia-container-toolkit.list apt-get update apt-get install -y nvidia-container-toolkit Configure Docker and Containerd to use NVIDIA Container Toolkit\n1 2 nvidia-ctk runtime configure --runtime=docker nvidia-ctk runtime configure --runtime=containerd Since we are running inside unprivileged containers, we need to set no-cgroups to true\n1 nvidia-ctk config --set nvidia-container-cli.no-cgroups --in-place Install NVIDIA Nsight Systems So we can profile applications running on the GPU.\n1 2 3 wget https://xxx/NsightSystems-linux-cli-public-2025.3.1.90-3582212.deb dpkg -i NsightSystems-linux-cli-public-2025.3.1.90-3582212.deb rm -f NsightSystems-linux-cli-public-2025.3.1.90-3582212.deb Functionality Tests NCCL 1 2 3 4 git clone https://github.com/NVIDIA/nccl-tests.git cd nccl-tests git reset --hard 903918f # I only tested this commit. make P2P ON: NCCL_P2P_LEVEL=sys ./build/all_reduce_perf --minbytes 8 --maxbytes 128M --stepfactor 2 --ngpus 8 (change --ngpus accordingly)\nP2P OFF: NCCL_P2P_DISABLE=1 ./build/all_reduce_perf --minbytes 8 --maxbytes 128M --stepfactor 2 --ngpus 8 (change --ngpus accordingly)\nYou should see a bandwidth increase after P2P is on.\nCUDA P2P 1 2 3 4 5 git clone https://github.com/NVIDIA/cuda-samples.git git reset --hard 9c688d7 # I only tested this commit. cd cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest make ./p2pBandwidthLatencyTest You should see lower GPU-GPU latency, higher bandwidth if P2P is on.\nPrepare for Templating Check if there are unnecessary files and remove them.\n1 2 cd ls -l Remove history\n1 2 3 4 5 6 7 8 9 10 11 12 cd unset HISTFILE rm .*_history echo -n \u0026gt;/var/log/lastlog echo -n \u0026gt;/var/log/wtmp echo -n \u0026gt;/var/log/btmp journalctl --vacuum-time=1s rm -f /var/log/*.log Now, it\u0026rsquo;s ready for converting to a template. To give each student a new development container, just clone this template.\nOther Minor Settings RAID Card Write Back To achieve maximum write speed, you can enable write back mode on your RAID card, so data is written to cache first. This will significantly (yes, by A LOT, sometimes over 100x) improve random write speed on HDDs (SSDs are already very fast). If your RAID card has a battery, you don\u0026rsquo;t need to worry about data integrity in case a power failure.\nWrite cache policy:\nWrite Throuh: no cache Write Back: cached if battery backup is present on the RAID card Always Write Back: always cached Obstacles Only 255 Cores are Recognized - x2APIC I noticed that I have one CPU offline. It should recognize 256 CPUs but I only have 255 CPUs.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 # lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 256 On-line CPU(s) list: 0-254 Off-line CPU(s) list: 255 \u0026lt;--------------- one CPU is offline ----------- Vendor ID: AuthenticAMD Model name: AMD EPYC 7763 64-Core Processor CPU family: 25 Model: 1 Thread(s) per core: 2 Core(s) per socket: 64 Socket(s): 2 After some digging, I learned that I need to enable x2APIC on both the motherboard and kernel to have more than 255 CPU cores recognized.\nIn BIOS Local APIC Mode should already be x2APIC .\nNow I realized it can be because I disabled IOMMU for PCIe P2P before. It turns out that disableing IOMMU can limit the number of available logical cores to 255. The reason is that the Linux kernel disables x2APIC in this case and falls back to APIC, which can only enumerate a maximum of 255 (logical) cores.\nI will keep IOMMU disabled because I need PCIe P2P to function (I don\u0026rsquo;t want to deal with P2P with IOMMU on). Just forget about the 256th core :p.\n","date":"2025-08-16T20:49:00+08:00","permalink":"https://charlie0129.github.io/blog/p/proxmox-ve-shared-gpu-installation-guide/","title":"Proxmox VE Shared GPU Installation Guide"},{"content":"启用 SSH 没SSH就别想着用Terminal了，赶紧去开了。\n去绿联NAS的Web管理界面，控制面板 -\u0026gt; 终端机 -\u0026gt; SSH ，记得关闭自动关闭功能。\n配置 root 用户 SSH 登录 用你的NAS用户名SSH到NAS上ssh yourname@nas-ip\n你需要先有一个SSH密钥对，如果你还没有，可以用 ssh-keygen -t rsa 生成一个。\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 # 切root sudo su # 打开 root SSH 登录（仅允许密钥登录） # 注意，别去改 /etc/ssh/sshd_config ，因为绿联NAS的系统会保护这个文件，改了重启就失效了。 echo \u0026#39;PermitRootLogin prohibit-password\u0026#39; \u0026gt;\u0026gt; /etc/ssh/sshd_config.d/A0-root-login.conf # 应用修改 systemctl restart ssh # 将你的公钥添加到 /root/.ssh/authorized_keys 中 mkdir -p /root/.ssh chmod 700 /root/.ssh echo \u0026#39;your-public-key\u0026#39; \u0026gt;\u0026gt; /root/.ssh/authorized_keys chmod 600 /root/.ssh/authorized_keys 配置你本机的 ~/.ssh/config 文件使用对应的密钥登录NAS：\n1 2 3 4 Host nas HostName nas-ip User root IdentityFile ~/.ssh/\u0026lt;private-key\u0026gt; 安装包管理器 Opkg 那么有人会问，你为什么要装包管理器，你怎么不用自带的 apt？UGOS Pro不是基于Debian的么？其实最主要的原因是避免干扰内置预装的package。我举个例子，你想安装git，好，那你得apt update吧，update完拉过来了最新的package index，然后刚好这个新的git依赖新版的libcurl，你装的时候给他一起升级了，但是绿联自带了不少预装的package，这些package可能是根据老版本libcurl编译的。正常的debian你upgrade一下就完事了，但是UGOS作为一个定制的Debian，不建议你这么做（其实要是你尝试去upgrade，你会发现不让你upgrade，绿联已经做了保护了，虽然你可以绕过就是）。\nOK，我们解释清楚了为什么不用apt。那么用什么呢？我的建议是一个轻量级给嵌入式设备用的包管理器，用OpenWRT的可能很熟悉了，就是opkg。你想重量级一点，也可以用Homebrew/Linuxbrew（不过你得先装git才能装Homebrew，但是git又没有，现在还没包管理器也没得装git，循环依赖了属于是），也可以用Nix（我觉得Nix太重了，除非你真的需要Nix的特性，不然就算了）。\n这里用 Opkg 做演示。你需要用root用户SSH到NAS上ssh root@nas-ip。\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 # 下载 opkg 安装脚本 wget https://mirrors.nju.edu.cn/entware/x64-k3.2/installer/generic.sh -O opkg-install.sh # 换源 1 sed -i \u0026#39;s|http://bin.entware.net|https://mirrors.nju.edu.cn/entware|g\u0026#39; opkg-install.sh # 换源 2 sed -i \u0026#39;s|-O /opt/etc/opkg.conf|-O /opt/etc/opkg.conf \\\u0026amp;\\\u0026amp; sed -i \u0026#34;s,http://bin.entware.net,https://mirrors.nju.edu.cn/entware,g\u0026#34; /opt/etc/opkg.conf|g\u0026#39; opkg-install.sh # 执行 opkg 安装脚本 /bin/sh opkg-install.sh # 临时设置 PATH ，使得 opkg 命令可用 export PATH=/opt/bin:/opt/sbin:/opt/usr/bin:$PATH # 设置 PATH ，使得 opkg 命令可用。如果你用 zsh，记得改成 ~/.zshrc 。 # 或者如果你用我的 dotfiles，现在可以先不做这一步，你可以看后续我配置 zsh 的时候怎么做的。 echo \u0026#39;export PATH=/opt/bin:/opt/sbin:/opt/usr/bin:$PATH\u0026#39; \u0026gt;\u0026gt; ~/.bashrc 安装常用Package 这时有包管理器了，而且是与NAS系统隔离的，你可以随意安装了。以下是一些常用的包：\n1 2 3 4 5 6 # 注意 vim 需要安装两个包，vim-full 和 vim-runtime opkg install vim-full vim-runtime # 终于能装 git 了，注意需要 git-http opkg install git git-http # 其他常见工具 opkg install htop sysstat zsh 配置 ZSH 如果你想要一个更好的终端体验，建议使用 ZSH 和比较好的配置。以下是使用我的 dotfiles 来配置 ZSH 的步骤：\n注意，你需要在你以后会经常使用的用户中配置 ZSH 。 我这里都是用的 root 用户，我对 Linux 比较熟悉了所以我并不怕搞坏东西。 如果你不熟悉 Linux，建议切换回自己用户配置 ZSH 。 如果找不到 git ，记得参考 opkg 安装中最后一步重新设置 PATH 。\n1 2 3 4 5 6 7 8 9 10 # 先设置代理，等会要从 GitHub 上下载文件 export https_proxy=http://\u0026lt;your-proxy-ip\u0026gt;:\u0026lt;your-proxy-port\u0026gt; # 克隆我的 dotfiles 仓库 cd git clone https://github.com/charlie0129/dotfiles.git cd dotfiles ./bootstrap.sh -f zsh # 询问 Do you need to use a proxy [y/n] 的时候选择 n （因为前面 set 过 proxy 了） # 询问 Change login shell of root to /opt/bin/zsh 的时候选择 y （不然呢） 将 Opkg 的 bin 和 sbin 目录添加到 ZSH 的 PATH 中：\n1 vim ~/dotfiles/env/custom.sh 1 2 3 4 5 6 7 8 9 # This list is inserted before PATH PATH_BEFORE=( # custom bin in this repo, i.e. bin/custom $HOME/dotfiles/bin/custom # Opkg bin and sbin directories /opt/bin /opt/sbin /opt/usr/bin ) 效果\n优化 Zram 需要使用 root 用户\n绿联默认的 zram 不够激进，关了它。我们要用 zstd 算法和一半的内存来做zram，榨干内存。\n1 2 3 4 5 6 7 # 先设置代理，等会要从 GitHub 上下载文件 export https_proxy=http://\u0026lt;your-proxy-ip\u0026gt;:\u0026lt;your-proxy-port\u0026gt; git clone --depth=1 https://github.com/foundObjects/zram-swap.git cd zram-swap ./install.sh cd .. rm -rf zram-swap 配置使用 zstd 算法（因为 Debian 13 里面默认 zram 不给 lzo-rle 算法了，lz4 的压缩率又不行，所以用 zstd ）：\n1 2 sed -i \u0026#39;s/_zram_algorithm=.*/_zram_algorithm=\u0026#34;zstd\u0026#34;/g\u0026#39; /etc/default/zram-swap systemctl restart zram-swap 为 zram 配置合适的 sysctl ，较大的 swappiness 值可以不活跃的页面更快地被交换出去，让内存留给更有用的页面。因为 sysctl.conf 里面绿联已经设置了 vm.swappiness 如果放在 /etc/sysctl.d/ 里面会被覆盖掉，所以直接放在 sysctl.conf 里面。\n1 2 3 4 5 6 7 echo \u0026#39;# ZRAM BEGIN\u0026#39; \u0026gt;\u0026gt; /etc/sysctl.conf echo \u0026#39;vm.swappiness = 180\u0026#39; \u0026gt;\u0026gt; /etc/sysctl.conf echo \u0026#39;vm.watermark_boost_factor = 0\u0026#39; \u0026gt;\u0026gt; /etc/sysctl.conf echo \u0026#39;vm.watermark_scale_factor = 125\u0026#39; \u0026gt;\u0026gt; /etc/sysctl.conf echo \u0026#39;vm.page-cluster = 0\u0026#39; \u0026gt;\u0026gt; /etc/sysctl.conf echo \u0026#39;# ZRAM END\u0026#39; \u0026gt;\u0026gt; /etc/sysctl.conf sysctl --system 能看到内存大小 1.5 倍的zram swap即成功\n1 2 3 $ swapon NAME TYPE SIZE USED PRIO /dev/zram0 partition 93.8G 0B 15 安装 Docker 先去应用中心安装 Docker 。默认绿联的Docker 配置比较烂，而且还会在根目录下留下一个 /daemon.json 的文件夹（WTF？），一看就是绿联的安装脚本写错了。\n1 2 # 删除错误的 daemon.json 文件夹 rmdir /daemon.json 将以下json写入 /etc/docker/daemon.json：\n1 2 3 4 5 6 7 8 9 10 11 { \u0026#34;log-opts\u0026#34;: { \u0026#34;max-size\u0026#34;: \u0026#34;1m\u0026#34; }, \u0026#34;experimental\u0026#34;: true, \u0026#34;metrics-addr\u0026#34;: \u0026#34;0.0.0.0:8132\u0026#34;, \u0026#34;data-root\u0026#34;: \u0026#34;\u0026lt;your-data-root\u0026gt;\u0026#34;, \u0026#34;registry-mirrors\u0026#34;: [ \u0026#34;https://xxxxxx.com/dockerhub/\u0026#34;, ] } log-opts：限制日志大小，防止日志占满磁盘。 experimental：启用实验性功能。 metrics-addr：设置可观测端口。 data-root：设置 Docker 数据目录，建议查看你之前的 /etc/docker/daemon.json 中的配置，不要动。 registry-mirrors：设置 Docker 镜像加速器，建议使用国内的镜像源。 可观测配置 Grafana: 用于可视化和监控。 Victoria Metrics: 用于指标存储与查询。 Alloy: 用于容器日志采集。 Cadvisor: 用于容器指标采集。 Loki: 用于日志存储与查询。 Node Exporter: 用于主机指标采集。 Smartctl Exporter: 用于硬盘SMART指标采集。 Intel PCM: 用于CPU性能监控。 效果：TODO\n参考 Docker Compose （镜像版本可以适当升级，记得更换volume中数据存储位置）：\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 name: o11y services: grafana: container_name: grafana image: grafana/grafana:12.1.0 restart: unless-stopped mem_limit: 1G user: 0:0 networks: - o11y # 我使用了 Traefik 作为反向代理，所以不需要暴露端口 # 如果你没有使用 Traefik，可以取消注释以下端口映射 # ports: # - 3000:3000 healthcheck: test: wget --no-verbose --tries=1 --spider http://localhost:3000/api/health volumes: - /volume2/docker/data/grafana:/var/lib/grafana environment: TZ: Asia/Shanghai GF_SERVER_ENABLE_GZIP: true # 如果你使用 Traefik 作为反向代理，可以添加以下标签 labels: - \u0026#34;traefik.enable=true\u0026#34; - \u0026#34;traefik.http.routers.grafana.entrypoints=web\u0026#34; - \u0026#34;traefik.http.routers.grafana.rule=Host(`grafana.example.com`)\u0026#34; - \u0026#34;traefik.http.routers.grafana.service=grafana-secure\u0026#34; # - \u0026#34;traefik.http.routers.grafana.middlewares=grafana-https-redirect\u0026#34; # - \u0026#34;traefik.http.middlewares.grafana-https-redirect.redirectscheme.scheme=https\u0026#34; - \u0026#34;traefik.http.routers.grafana-secure.tls=true\u0026#34; - \u0026#34;traefik.http.routers.grafana-secure.tls.certresolver=cloudflare\u0026#34; - \u0026#34;traefik.http.routers.grafana-secure.entrypoints=websecure\u0026#34; - \u0026#34;traefik.http.routers.grafana-secure.rule=Host(`grafana.example.com`)\u0026#34; - \u0026#34;traefik.http.services.grafana-secure.loadbalancer.server.port=3000\u0026#34; vm: container_name: vm image: victoriametrics/victoria-metrics:v1.122.0 restart: unless-stopped mem_limit: 4G user: 0:0 networks: - o11y # ports: # - 8428:8428 healthcheck: test: wget --no-verbose --tries=1 --spider http://localhost:8428/ || exit 1 command: - -httpListenAddr=0.0.0.0:8428 - -promscrape.config=/etc/victoriametrics/scrape.yml - -storageDataPath=/var/victoriametrics - -retentionPeriod=20y - -inmemoryDataFlushInterval=300s # 减少写盘 extra_hosts: - \u0026#39;host.docker.internal:host-gateway\u0026#39; - \u0026#39;d48t:host-gateway\u0026#39; volumes: - ./vm:/etc/victoriametrics # Scrape 配置 - /volume2/docker/data/vm:/var/victoriametrics alloy: container_name: alloy image: grafana/alloy:v1.10.0 restart: unless-stopped user: 0:0 networks: - o11y mem_limit: 512M # ports: # - 12345:12345 command: - run - --disable-reporting - --server.http.listen-addr=0.0.0.0:12345 - --storage.path=/var/lib/alloy/data - /etc/alloy/config.alloy volumes: - /var/run/docker.sock:/var/run/docker.sock:ro - /tmp/alloy:/var/lib/alloy/data # 这玩意巨能写，SSD会写爆，给他放内存里去。绿联的 /tmp 是内存盘。 - ./alloy/config.alloy:/etc/alloy/config.alloy cadvisor: container_name: cadvisor image: gcr.io/cadvisor/cadvisor:v0.53.0 mem_limit: 256M restart: unless-stopped user: 0:0 # ports: # - 8080:8080 networks: - o11y volumes: - /:/rootfs:ro - /var/run:/var/run:ro - /sys:/sys:ro - /var/lib/docker/:/var/lib/docker:ro - /dev/disk/:/dev/disk:ro privileged: true devices: - /dev/kmsg command: - --store_container_labels=false - --docker_only=true - --housekeeping_interval=30s loki: container_name: loki image: grafana/loki:3.5.3 restart: unless-stopped user: 0:0 networks: o11y: ipv4_address: \u0026#34;172.26.195.254\u0026#34; # 固定 IP，方便 Docker 的 Loki plugin 用（虽然我们这里用的 Alloy 其实并不需要） # 如果你没有使用 Traefik 作为反向代理，可以取消注释以下端口映射。 # ports: # - 3100:3100 mem_limit: 4G command: - \u0026#34;-config.file=/etc/loki/config.yml\u0026#34; volumes: - ./loki:/etc/loki:ro - /volume2/docker/data/loki:/loki node-exporter: container_name: node-exporter image: prom/node-exporter:v1.9.1 restart: unless-stopped mem_limit: 256M user: 0:0 privileged: true # ports: # - 9100:9100 command: - --path.rootfs=/host - --web.listen-address=172.17.0.1:9100 network_mode: host # networks: # - o11y pid: host volumes: - \u0026#39;/:/host:ro,rslave\u0026#39; smartctl-exporter: container_name: smartctl-exporter image: prometheuscommunity/smartctl-exporter:v0.14.0 restart: unless-stopped mem_limit: 512M user: 0:0 # ports: # - 192.168.91.1:9633:9633 networks: - o11y privileged: true pcm: container_name: pcm image: opcm/pcm mem_limit: 256M restart: unless-stopped user: 0:0 privileged: true # ports: # - 192.168.91.1:9738:9738 networks: - o11y networks: o11y: name: o11y driver: bridge ipam: config: - subnet: \u0026#34;172.26.195.0/24\u0026#34; 参考 Victoria Metrics 的 Scrape 配置：\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 # ./vm/scrape.yml global: scrape_interval: 15s scrape_configs: - job_name: victoria-metrics static_configs: - targets: - vm:8428 # 如果你有 Traefik # - job_name: traefik # static_configs: # - targets: # - traefik:8082 - job_name: node-exporter static_configs: - targets: - d48t:9100 - job_name: cadvisor static_configs: - targets: - cadvisor:8080 - job_name: loki static_configs: - targets: - loki:3100 - job_name: grafana static_configs: - targets: - grafana:3000 - job_name: smartctl-exporter static_configs: - targets: - smartctl-exporter:9633 - job_name: pcm static_configs: - targets: - pcm:9738 参考 Loki 配置：\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 # ./loki/config.yml auth_enabled: false server: http_listen_port: 3100 common: instance_addr: 127.0.0.1 path_prefix: /loki storage: filesystem: chunks_directory: /loki/chunks rules_directory: /loki/rules replication_factor: 1 ring: kvstore: store: inmemory ingester: chunk_encoding: lz4 # 比默认的snappy压缩率高 chunk_target_size: 8388608 # 8M max_chunk_age: 48h chunk_idle_period: 12h limits_config: max_query_lookback: 672h # 28 days retention_period: 672h # 28 days schema_config: configs: - from: 2020-10-24 store: tsdb object_store: filesystem schema: v13 index: prefix: index_ period: 24h ruler: alertmanager_url: http://localhost:9093 参考 Grafana Alloy 配置：\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 # ./alloy/config.alloy logging { level = \u0026#34;info\u0026#34; format = \u0026#34;logfmt\u0026#34; } // Discover Docker containers and extract metadata. discovery.docker \u0026#34;logs_integrations_docker\u0026#34; { host = \u0026#34;unix:///var/run/docker.sock\u0026#34; refresh_interval = \u0026#34;5s\u0026#34; } // Define a relabeling rule to create a service name from the container name. discovery.relabel \u0026#34;logs_integrations_docker\u0026#34; { targets = [] rule { target_label = \u0026#34;job\u0026#34; replacement = \u0026#34;integrations/docker\u0026#34; } rule { target_label = \u0026#34;instance\u0026#34; replacement = \u0026#34;d48t\u0026#34; // constants.hostname } rule { source_labels = [\u0026#34;__meta_docker_container_name\u0026#34;] regex = \u0026#34;/(.*)\u0026#34; target_label = \u0026#34;container\u0026#34; } rule { source_labels = [\u0026#34;__meta_docker_container_log_stream\u0026#34;] target_label = \u0026#34;stream\u0026#34; } } // Configure a loki.source.docker component to collect logs from Docker containers. loki.source.docker \u0026#34;logs_integrations_docker\u0026#34; { host = \u0026#34;unix:///var/run/docker.sock\u0026#34; targets = discovery.docker.logs_integrations_docker.targets relabel_rules = discovery.relabel.logs_integrations_docker.rules forward_to = [loki.write.local.receiver] refresh_interval = \u0026#34;15s\u0026#34; } loki.write \u0026#34;local\u0026#34; { endpoint { url = \u0026#34;http://loki:3100/loki/api/v1/push\u0026#34; } } ","date":"2025-07-27T16:45:00+08:00","permalink":"https://charlie0129.github.io/blog/p/ugreen-nas-terminal-setup/","title":"(Chinese-Only) 一个Linux用户的绿联NAS配置指南"},{"content":"If you want see the test results directly, please jump to the Test the unlocked performance section.\nWhat is P2P? From NVIDIA: GPUDirect Peer to Peer enables GPU-to-GPU copies as well as loads and stores directly over the memory fabric (PCIe, NVLink). GPUDirect Peer to Peer is supported natively by the CUDA Driver.\nAs you can see, GPUDirect P2P allows for direct memory access between GPUs, bypassing the CPU and system memory. This can lead to significant performance improvements in multi-GPU systems. For example, in a multi-GPU LLM inference system, the data can be directly transferred between GPUs without involving the CPU, reducing latency and improving throughput.\nWhy is P2P disabled on RTX4090? Easy, NVIDIA wants to make money. They want you to buy the more expensive Tesla GPUs if you want P2P.\nTest machine configuration CPU: 2 x AMD EPYC 7542 (32C 64T, 225W TDP, 2.9GHz base, 3.4GHz boost) MEM: 2 x 8 x 32GB 3200MT/s DDR4 ECC GPU: 8 x NVIDIA GeForce RTX 4090 (24GB GDDR6X, 450W TDP) Topology (ideally, the GPUs should be behind a PCIe switch, but our test machine doesn\u0026rsquo;t have one):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS 24-31,88-95 3 N/A GPU1 SYS X SYS SYS SYS SYS SYS SYS SYS SYS SYS SYS 16-23,80-87 2 N/A GPU2 SYS SYS X SYS SYS SYS SYS SYS SYS SYS SYS SYS 8-15,72-79 1 N/A GPU3 SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS SYS 0-7,64-71 0 N/A GPU4 SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS 56-63,120-127 7 N/A GPU5 SYS SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS 48-55,112-119 6 N/A GPU6 SYS SYS SYS SYS SYS SYS X SYS PHB PHB PHB PHB 40-47,104-111 5 N/A GPU7 SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS 32-39,96-103 4 N/A NIC0 SYS SYS SYS SYS SYS SYS PHB SYS X PIX PHB PHB NIC1 SYS SYS SYS SYS SYS SYS PHB SYS PIX X PHB PHB NIC2 SYS SYS SYS SYS SYS SYS PHB SYS PHB PHB X PIX NIC3 SYS SYS SYS SYS SYS SYS PHB SYS PHB PHB PIX X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 Test the original performance PCI BAR Only 256MB of BAR memory is available. Note that I did enabled Resizable BAR in the BIOS (not my fault).\n1 2 3 4 5 # nvidia-smi -q | grep -i bar -A 3 BAR1 Memory Usage Total : 256 MiB Used : 1 MiB Free : 255 MiB NCCL Tests Building NCCL Tests\n1 2 3 git clone --depth=1 https://github.com/NVIDIA/nccl-tests.git cd nccl-tests make -j$(nproc) Runnning:\nAround 14.5GB/s of bandwidth. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 # ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 8 # nThread 1 nGpus 8 minBytes 8 maxBytes 134217728 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0 # # Using devices # Rank 0 Group 0 Pid 1756513 on \u0026lt;REDACTED\u0026gt; device 0 [0x01] NVIDIA GeForce RTX 4090 # Rank 1 Group 0 Pid 1756513 on \u0026lt;REDACTED\u0026gt; device 1 [0x25] NVIDIA GeForce RTX 4090 # Rank 2 Group 0 Pid 1756513 on \u0026lt;REDACTED\u0026gt; device 2 [0x41] NVIDIA GeForce RTX 4090 # Rank 3 Group 0 Pid 1756513 on \u0026lt;REDACTED\u0026gt; device 3 [0x61] NVIDIA GeForce RTX 4090 # Rank 4 Group 0 Pid 1756513 on \u0026lt;REDACTED\u0026gt; device 4 [0x81] NVIDIA GeForce RTX 4090 # Rank 5 Group 0 Pid 1756513 on \u0026lt;REDACTED\u0026gt; device 5 [0xa1] NVIDIA GeForce RTX 4090 # Rank 6 Group 0 Pid 1756513 on \u0026lt;REDACTED\u0026gt; device 6 [0xc1] NVIDIA GeForce RTX 4090 # Rank 7 Group 0 Pid 1756513 on \u0026lt;REDACTED\u0026gt; device 7 [0xe1] NVIDIA GeForce RTX 4090 # # out-of-place in-place # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 8 2 float sum -1 38.83 0.00 0.00 0 40.43 0.00 0.00 0 16 4 float sum -1 38.73 0.00 0.00 0 38.59 0.00 0.00 0 32 8 float sum -1 41.85 0.00 0.00 0 42.55 0.00 0.00 0 64 16 float sum -1 42.55 0.00 0.00 0 42.65 0.00 0.00 0 128 32 float sum -1 42.42 0.00 0.01 0 42.88 0.00 0.01 0 256 64 float sum -1 42.44 0.01 0.01 0 42.53 0.01 0.01 0 512 128 float sum -1 42.58 0.01 0.02 0 42.68 0.01 0.02 0 1024 256 float sum -1 44.04 0.02 0.04 0 42.39 0.02 0.04 0 2048 512 float sum -1 42.91 0.05 0.08 0 42.61 0.05 0.08 0 4096 1024 float sum -1 43.18 0.09 0.17 0 42.69 0.10 0.17 0 8192 2048 float sum -1 44.02 0.19 0.33 0 42.96 0.19 0.33 0 16384 4096 float sum -1 43.10 0.38 0.67 0 42.99 0.38 0.67 0 32768 8192 float sum -1 44.18 0.74 1.30 0 43.63 0.75 1.31 0 65536 16384 float sum -1 45.66 1.44 2.51 0 45.28 1.45 2.53 0 131072 32768 float sum -1 67.27 1.95 3.41 0 54.86 2.39 4.18 0 262144 65536 float sum -1 124.8 2.10 3.68 0 121.1 2.17 3.79 0 524288 131072 float sum -1 196.9 2.66 4.66 0 200.5 2.61 4.58 0 1048576 262144 float sum -1 222.8 4.71 8.24 0 226.1 4.64 8.12 0 2097152 524288 float sum -1 373.4 5.62 9.83 0 358.6 5.85 10.23 0 4194304 1048576 float sum -1 580.8 7.22 12.64 0 592.0 7.09 12.40 0 8388608 2097152 float sum -1 1041.8 8.05 14.09 0 1040.2 8.06 14.11 0 16777216 4194304 float sum -1 1972.8 8.50 14.88 0 1979.5 8.48 14.83 0 33554432 8388608 float sum -1 3905.6 8.59 15.04 0 3919.0 8.56 14.98 0 67108864 16777216 float sum -1 7907.1 8.49 14.85 0 7905.3 8.49 14.86 0 134217728 33554432 float sum -1 16253 8.26 14.45 0 16227 8.27 14.47 0 # Out of bounds values : 0 OK # Avg bus bandwidth : 4.85265 # P2P Bandwidth and Latency Building:\n1 2 3 git clone --depth=1 https://github.com/NVIDIA/cuda-samples.git cd cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest make -j$(nproc) Running:\nAbout 21GB/s bidirectional bandwidth 10us+ GPU-GPU latency 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 # ./p2pBandwidthLatencyTest P2P Connectivity Matrix D\\D 0 1 2 3 4 5 6 7 0 1 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 2 0 0 1 0 0 0 0 0 3 0 0 0 1 0 0 0 0 4 0 0 0 0 1 0 0 0 5 0 0 0 0 0 1 0 0 6 0 0 0 0 0 0 1 0 7 0 0 0 0 0 0 0 1 Unidirectional P2P=Disabled Bandwidth Matrix (GB/s) D\\D 0 1 2 3 4 5 6 7 0 909.49 18.77 19.56 20.23 18.72 19.29 19.34 19.83 1 19.95 919.12 18.93 20.18 18.83 19.23 19.24 19.80 2 19.87 19.55 919.66 20.27 18.82 19.28 19.29 19.77 3 19.92 19.48 19.56 918.58 18.82 19.23 19.19 19.80 4 19.88 20.58 19.74 20.54 920.15 18.38 18.89 18.71 5 20.04 20.19 19.80 20.37 18.75 921.29 18.88 19.08 6 20.04 20.11 19.71 20.60 18.77 18.58 920.74 18.62 7 20.02 20.21 19.73 20.33 18.37 18.74 18.89 921.29 Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s) D\\D 0 1 2 3 4 5 6 7 0 910.02 18.83 19.55 20.27 18.76 19.36 19.26 19.83 1 19.95 920.20 18.92 20.19 18.83 19.26 19.21 19.80 2 19.82 19.61 919.66 20.28 18.86 19.29 19.34 19.75 3 19.91 19.47 19.60 920.20 18.83 19.26 19.16 19.81 4 19.78 20.43 19.75 20.56 920.25 18.60 18.85 18.75 5 20.02 20.24 19.76 20.45 18.80 922.37 18.89 19.15 6 20.06 20.12 19.75 20.59 18.80 18.58 921.25 18.61 7 20.02 20.16 19.80 20.38 18.41 18.72 18.87 920.83 Bidirectional P2P=Disabled Bandwidth Matrix (GB/s) D\\D 0 1 2 3 4 5 6 7 0 916.15 21.03 21.53 21.54 20.69 21.02 21.10 21.06 1 21.39 920.74 21.59 21.52 20.55 21.02 20.50 21.18 2 21.15 21.48 921.83 21.20 20.58 21.11 20.98 21.09 3 21.38 21.67 21.42 922.65 20.53 21.05 20.65 21.21 4 20.86 20.89 20.86 20.95 923.69 19.99 20.11 19.97 5 20.90 20.97 21.00 21.02 20.22 922.10 20.15 20.30 6 20.94 20.62 20.89 20.61 19.98 20.07 923.11 19.99 7 21.07 21.13 21.10 21.22 20.00 20.29 20.11 923.46 Bidirectional P2P=Enabled Bandwidth Matrix (GB/s) D\\D 0 1 2 3 4 5 6 7 0 917.23 20.86 21.49 21.56 20.63 20.99 21.05 21.13 1 21.30 922.37 21.61 21.56 20.53 21.08 20.49 21.15 2 21.15 21.47 922.36 21.16 20.50 21.06 20.94 21.10 3 21.39 21.60 21.39 922.92 20.63 21.09 20.69 21.22 4 20.89 20.86 20.81 20.89 923.46 20.12 20.05 20.02 5 20.88 20.98 20.99 20.87 20.20 923.65 20.12 20.39 6 21.05 20.63 20.89 20.67 20.07 20.05 922.92 20.02 7 21.01 21.12 21.06 21.19 20.01 20.31 20.13 923.16 P2P=Disabled Latency Matrix (us) GPU 0 1 2 3 4 5 6 7 0 1.54 20.26 11.68 11.60 12.65 15.80 12.19 11.75 1 11.60 1.48 11.58 11.66 16.03 12.13 12.55 14.59 2 20.27 11.59 1.43 20.08 13.17 17.14 16.31 11.41 3 11.50 11.59 11.51 1.47 13.11 16.14 14.04 11.61 4 12.53 14.27 12.75 12.65 1.51 12.47 12.49 12.41 5 11.88 11.73 11.65 11.63 12.28 1.42 14.95 15.40 6 12.27 12.63 12.42 12.68 12.25 12.30 1.47 11.89 7 11.46 16.67 11.44 11.47 12.30 13.33 13.57 1.41 CPU 0 1 2 3 4 5 6 7 0 4.04 12.23 10.01 9.91 10.88 10.78 10.60 11.29 1 9.79 3.15 9.85 10.06 10.77 10.56 10.50 10.48 2 10.04 9.63 3.24 9.94 10.86 10.65 10.70 10.63 3 9.75 9.61 9.85 3.26 10.92 10.82 10.77 10.67 4 10.50 10.26 10.61 10.56 3.57 11.55 11.48 11.26 5 10.52 10.07 10.40 10.48 11.42 3.52 11.32 11.24 6 10.24 10.20 10.44 10.47 11.42 11.29 3.54 11.35 7 10.26 10.09 10.50 10.41 11.47 11.31 11.36 3.53 P2P=Enabled Latency (P2P Writes) Matrix (us) GPU 0 1 2 3 4 5 6 7 0 1.54 11.67 11.75 20.27 14.49 15.77 16.95 15.91 1 11.67 1.48 11.44 11.59 15.22 12.17 13.20 13.18 2 11.58 20.27 1.43 11.50 14.60 13.73 13.21 11.38 3 11.58 20.06 12.12 1.47 13.16 16.17 14.67 11.67 4 12.88 12.78 13.92 12.58 1.50 12.46 12.59 12.50 5 11.48 11.93 11.78 11.88 19.18 1.42 11.59 11.40 6 11.95 11.70 14.90 14.00 12.42 11.41 1.46 11.63 7 12.96 11.45 11.60 11.53 20.55 11.32 12.94 1.40 CPU 0 1 2 3 4 5 6 7 0 3.21 9.61 9.86 9.79 10.69 10.48 10.55 10.44 1 9.67 3.12 9.81 9.81 10.64 10.46 10.66 10.43 2 9.89 9.62 3.20 9.98 10.96 10.67 10.75 10.72 3 9.80 9.65 9.89 3.20 10.86 10.78 10.78 10.61 4 10.41 10.24 10.60 10.50 3.52 11.34 11.49 11.28 5 10.27 10.14 11.93 14.06 14.53 4.55 16.99 19.67 6 20.68 20.61 20.39 20.63 20.86 14.35 5.21 14.46 7 9.89 9.66 9.85 9.85 10.81 10.62 10.68 3.41 How to unlock P2P on RTX4090? I encountered this legendary modified NVIDIA driver by tinygrad that unlocks P2P on RTX4090 GPUs which blows my mind. Let\u0026rsquo;s give it a try!\nDisable IOMMU If the following command returns an empty list, it means that the IOMMU is disabled, otherwise, you need to disable it. Consult your motherboard manual on how to disable IOMMU.\n1 ll /sys/class/iommu/ Enable Resizable BAR Enable Resizable BAR in the BIOS. It\u0026rsquo;s required for the modified NVIDIA driver to work. Consult your motherboard manual on how to enable Resizable BAR.\nUninstall the official NVIDIA driver We will install our own modified NVIDIA driver, so we need to uninstall the official NVIDIA driver first.\n1 2 nvidia-uninstall reboot Install the official NVIDIA driver without kernel modules Visit the modified NVIDIA driver by tinygrad. Check out the branches with p2p in the name. There are multiple versions. You should choose one that fits. For example, I am using the 550.90.07-p2p branch. Remember the version number 550.90.07, you will need it later.\nDownload the official driver with the same version number as you chose (550.90.07 in my case) from the NVIDIA website. The driver I downloaded is NVIDIA-Linux-x86_64-550.90.07.run.\nInstall the official driver without the kernel modules because we will install our own modified kernel modules later.\n1 ./NVIDIA-Linux-x86_64-550.90.07.run --no-kernel-modules Install the modified kernel modules Clone the modified NVIDIA driver by tinygrad and install. Remember to replace the branch name with the one you chose.\n1 2 3 4 5 6 7 8 9 git clone --depth=1 -b 550.90.07-p2p https://github.com/tinygrad/open-gpu-kernel-modules.git cd open-gpu-kernel-modules # Just in case rmmod nvidia_drm nvidia_modeset nvidia_uvm nvidia # Build and install the modified kernel modules make modules -j$(nproc) make modules_install depmod nvidia-smi -pm 1 Test the unlocked performance PCI BAR Now the BAR memory is fully available.\n256MB -\u0026gt; 32GB. 1 2 3 4 5 # nvidia-smi -q | grep -i bar -A 3 BAR1 Memory Usage Total : 32768 MiB Used : 24211 MiB Free : 8557 MiB NCCL Tests 14.47GB/s -\u0026gt; 20.64GB/s: 42% improvement 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 # nThread 1 nGpus 8 minBytes 8 maxBytes 134217728 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0 # # Using devices # Rank 0 Group 0 Pid 33895 on \u0026lt;REDACTED\u0026gt; device 0 [0x01] NVIDIA GeForce RTX 4090 # Rank 1 Group 0 Pid 33895 on \u0026lt;REDACTED\u0026gt; device 1 [0x25] NVIDIA GeForce RTX 4090 # Rank 2 Group 0 Pid 33895 on \u0026lt;REDACTED\u0026gt; device 2 [0x41] NVIDIA GeForce RTX 4090 # Rank 3 Group 0 Pid 33895 on \u0026lt;REDACTED\u0026gt; device 3 [0x61] NVIDIA GeForce RTX 4090 # Rank 4 Group 0 Pid 33895 on \u0026lt;REDACTED\u0026gt; device 4 [0x81] NVIDIA GeForce RTX 4090 # Rank 5 Group 0 Pid 33895 on \u0026lt;REDACTED\u0026gt; device 5 [0xa1] NVIDIA GeForce RTX 4090 # Rank 6 Group 0 Pid 33895 on \u0026lt;REDACTED\u0026gt; device 6 [0xc1] NVIDIA GeForce RTX 4090 # Rank 7 Group 0 Pid 33895 on \u0026lt;REDACTED\u0026gt; device 7 [0xe1] NVIDIA GeForce RTX 4090 # # out-of-place in-place # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 8 2 float sum -1 38.82 0.00 0.00 0 38.16 0.00 0.00 0 16 4 float sum -1 38.31 0.00 0.00 0 38.12 0.00 0.00 0 32 8 float sum -1 38.57 0.00 0.00 0 40.87 0.00 0.00 0 64 16 float sum -1 40.71 0.00 0.00 0 40.65 0.00 0.00 0 128 32 float sum -1 40.51 0.00 0.01 0 41.66 0.00 0.01 0 256 64 float sum -1 41.63 0.01 0.01 0 41.38 0.01 0.01 0 512 128 float sum -1 41.67 0.01 0.02 0 42.45 0.01 0.02 0 1024 256 float sum -1 41.70 0.02 0.04 0 41.67 0.02 0.04 0 2048 512 float sum -1 41.74 0.05 0.09 0 41.74 0.05 0.09 0 4096 1024 float sum -1 42.35 0.10 0.17 0 42.39 0.10 0.17 0 8192 2048 float sum -1 42.14 0.19 0.34 0 41.88 0.20 0.34 0 16384 4096 float sum -1 43.19 0.38 0.66 0 42.73 0.38 0.67 0 32768 8192 float sum -1 42.17 0.78 1.36 0 42.61 0.77 1.35 0 65536 16384 float sum -1 47.58 1.38 2.41 0 46.95 1.40 2.44 0 131072 32768 float sum -1 72.03 1.82 3.18 0 71.50 1.83 3.21 0 262144 65536 float sum -1 119.7 2.19 3.83 0 118.6 2.21 3.87 0 524288 131072 float sum -1 166.6 3.15 5.51 0 159.8 3.28 5.74 0 1048576 262144 float sum -1 216.1 4.85 8.49 0 217.2 4.83 8.45 0 2097152 524288 float sum -1 321.5 6.52 11.42 0 322.6 6.50 11.37 0 4194304 1048576 float sum -1 478.7 8.76 15.33 0 480.1 8.74 15.29 0 8388608 2097152 float sum -1 798.6 10.50 18.38 0 818.0 10.26 17.95 0 16777216 4194304 float sum -1 1472.8 11.39 19.94 0 1472.1 11.40 19.94 0 33554432 8388608 float sum -1 2870.3 11.69 20.46 0 2866.4 11.71 20.49 0 67108864 16777216 float sum -1 5705.0 11.76 20.59 0 5695.6 11.78 20.62 0 134217728 33554432 float sum -1 11382 11.79 20.64 0 11380 11.79 20.64 0 # Out of bounds values : 0 OK # Avg bus bandwidth : 6.11168 # P2P Bandwidth and Latency P2P is now available: see the 1s in the P2P Connectivity Matrix Bandwidth: 21.39GB/s -\u0026gt; 50.15GB/s: 134% improvement Latency: 11.67us -\u0026gt; 1.19us: 89% reduction 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 P2P Connectivity Matrix D\\D 0 1 2 3 4 5 6 7 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 3 1 1 1 1 1 1 1 1 4 1 1 1 1 1 1 1 1 5 1 1 1 1 1 1 1 1 6 1 1 1 1 1 1 1 1 7 1 1 1 1 1 1 1 1 Unidirectional P2P=Disabled Bandwidth Matrix (GB/s) D\\D 0 1 2 3 4 5 6 7 0 912.13 18.86 19.89 20.25 18.34 19.64 19.25 19.45 1 20.56 920.74 19.44 20.27 18.46 19.72 19.19 19.40 2 20.36 19.72 922.37 20.28 18.52 19.65 19.22 19.40 3 20.43 19.68 20.04 922.49 18.47 19.59 19.15 19.41 4 20.20 20.15 20.25 20.26 921.83 19.04 18.84 18.36 5 20.45 20.09 20.17 20.46 18.29 922.37 18.73 18.60 6 20.39 20.18 20.24 20.54 18.44 19.09 922.49 18.65 7 20.53 20.14 20.12 20.51 18.17 19.11 18.74 923.46 Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s) D\\D 0 1 2 3 4 5 6 7 0 913.74 25.60 25.85 25.85 21.36 22.41 21.74 22.48 1 25.78 938.91 25.79 25.79 20.70 22.39 22.53 21.63 2 25.70 25.95 939.00 25.78 21.42 21.59 22.21 22.16 3 25.60 25.93 25.53 938.42 22.14 22.30 21.70 22.45 4 22.41 21.61 22.35 22.49 940.70 25.69 25.96 25.90 5 21.34 22.38 21.72 22.41 25.77 939.57 25.95 25.87 6 20.57 22.27 22.32 21.60 25.75 26.01 941.73 25.83 7 21.32 21.52 22.39 22.48 25.51 25.83 25.75 938.44 Bidirectional P2P=Disabled Bandwidth Matrix (GB/s) D\\D 0 1 2 3 4 5 6 7 0 918.85 20.94 21.69 21.67 20.66 21.44 20.98 21.26 1 21.48 921.29 21.57 21.54 20.20 20.65 20.87 20.70 2 21.34 21.63 921.56 21.13 20.66 21.28 20.89 21.21 3 21.40 21.72 21.62 921.83 20.16 20.92 20.89 20.89 4 20.93 20.33 20.87 20.45 922.90 20.07 20.21 19.96 5 21.40 20.86 21.26 20.90 20.23 923.69 20.18 20.33 6 20.93 20.87 20.89 20.90 20.27 20.25 923.67 20.19 7 21.19 20.66 21.03 20.87 20.02 20.28 20.41 923.41 Bidirectional P2P=Enabled Bandwidth Matrix (GB/s) D\\D 0 1 2 3 4 5 6 7 0 916.45 50.10 50.46 50.29 39.90 41.85 40.83 41.68 1 50.15 919.60 50.19 50.22 38.93 41.79 41.87 40.82 2 49.91 50.42 920.74 50.53 40.19 40.94 41.86 41.61 3 50.08 50.48 50.35 919.66 40.22 41.69 40.92 41.78 4 41.60 40.94 41.86 41.93 919.93 49.80 50.36 50.22 5 40.25 41.68 39.96 41.64 49.75 921.83 50.20 50.66 6 39.59 41.58 41.83 40.95 49.89 50.43 921.56 50.59 7 40.36 40.86 41.76 41.82 49.96 50.43 50.28 921.01 P2P=Disabled Latency Matrix (us) GPU 0 1 2 3 4 5 6 7 0 1.50 11.60 20.17 20.18 13.08 11.27 14.58 19.33 1 11.68 1.38 11.74 20.17 12.60 11.39 14.35 14.59 2 11.33 11.34 1.31 11.50 16.96 20.53 16.53 20.53 3 11.51 11.51 11.51 1.37 17.86 11.37 15.80 20.54 4 11.47 11.79 11.44 12.57 1.38 12.24 12.93 11.73 5 11.48 11.79 11.64 11.64 14.40 1.44 13.46 14.35 6 13.59 15.05 13.35 12.52 12.06 11.67 1.40 12.34 7 19.32 19.95 11.39 11.44 12.87 12.23 14.05 1.38 CPU 0 1 2 3 4 5 6 7 0 3.28 10.21 10.15 9.83 10.92 10.75 10.88 10.71 1 10.07 3.14 9.81 9.56 10.58 10.41 10.61 10.35 2 10.13 9.68 3.28 9.62 10.71 10.56 10.73 10.48 3 9.81 9.46 9.68 3.13 10.60 10.41 10.52 10.35 4 10.60 10.20 10.40 10.16 3.48 11.19 11.37 11.07 5 10.37 10.02 10.29 10.04 11.10 3.49 11.24 11.00 6 10.51 10.20 10.35 10.11 11.34 11.09 3.47 11.16 7 10.40 10.11 10.30 10.06 11.17 11.07 11.22 3.42 P2P=Enabled Latency (P2P Writes) Matrix (us) GPU 0 1 2 3 4 5 6 7 0 1.48 1.24 1.26 1.25 1.41 1.41 1.36 1.42 1 1.19 1.38 1.18 1.18 1.29 1.42 1.29 1.26 2 1.21 1.20 1.31 1.19 1.30 1.34 1.44 1.41 3 1.21 1.19 1.17 1.38 1.37 1.41 1.30 1.44 4 1.69 1.69 1.69 1.64 1.38 1.50 1.59 1.56 5 1.64 1.60 1.60 1.62 1.50 1.43 1.56 1.48 6 1.59 1.53 1.57 1.54 1.48 1.44 1.39 1.44 7 1.58 1.58 1.57 1.56 1.48 1.47 1.45 1.37 CPU 0 1 2 3 4 5 6 7 0 3.27 2.84 2.76 2.74 2.80 2.76 2.76 2.75 1 2.80 3.13 2.66 2.74 2.70 2.70 2.68 2.69 2 2.83 2.80 3.26 2.85 2.85 2.78 2.77 2.76 3 2.78 2.69 2.70 3.24 2.69 2.70 2.98 2.67 4 3.18 3.11 3.10 3.09 3.55 3.12 3.09 3.08 5 3.12 3.04 3.10 3.05 3.06 3.50 3.08 3.04 6 3.20 3.11 3.18 3.11 3.11 3.16 3.57 3.08 7 3.12 3.06 3.08 3.12 3.08 3.07 3.04 3.58 How it works? By tinygrad. I am not an expert in such low-level hacking (but it\u0026rsquo;s soooo interesting and I want to learn). This part is from the modified NVIDIA driver by tinygrad (Thank you tinygrad). I just put it here for reference.\nNormally, P2P on NVIDIA cards uses MAILBOXP2P. This is some hardware interface designed to allow GPUs to transfer memory back in the days of small BAR. It is not present or disabled in hardware on the 4090s, and that\u0026rsquo;s why P2P doesn\u0026rsquo;t work. There was a bug in early versions of the driver that reported that it did work, and it was actually sending stuff on the PCIe bus. However, because the mailbox hardware wasn\u0026rsquo;t present, these copies wouldn\u0026rsquo;t go to the right place. You could even crash the system by doing something like torch.zeros(10000,10000).cuda().to(\u0026quot;cuda:1\u0026quot;)\nIn some 3090s and all 4090s, NVIDIA added large BAR support.\n1 2 3 4 5 6 7 8 9 10 11 12 13 tiny@tiny14:~$ lspci -s 01:00.0 -v 01:00.0 VGA compatible controller: NVIDIA Corporation AD102 [GeForce RTX 4090] (rev a1) (prog-if 00 [VGA controller]) Subsystem: Micro-Star International Co., Ltd. [MSI] Device 510b Physical Slot: 49 Flags: bus master, fast devsel, latency 0, IRQ 377 Memory at b2000000 (32-bit, non-prefetchable) [size=16M] Memory at 28800000000 (64-bit, prefetchable) [size=32G] Memory at 28400000000 (64-bit, prefetchable) [size=32M] I/O ports at 3000 [size=128] Expansion ROM at b3000000 [virtual] [disabled] [size=512K] Capabilities: \u0026lt;access denied\u0026gt; Kernel driver in use: nvidia Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia Notice how BAR1 is size 32G. In H100, they also added support for a PCIe mode that uses the BAR directly instead of the mailboxes, called BAR1P2P. So, what happens if we try to enable that on a 4090?\nWe do this by bypassing the HAL and calling a bunch of the GH100 methods directly. Methods like kbusEnableStaticBar1Mapping_GH100, which maps the entire VRAM into BAR1. This mostly just works, but we had to disable the use of that region in the MapAperture function for some reason. Shouldn\u0026rsquo;t matter.\n1 2 [ 3491.654009] NVRM: kbusEnableStaticBar1Mapping_GH100: Static bar1 mapped offset 0x0 size 0x5e9200000 [ 3491.793389] NVRM: kbusEnableStaticBar1Mapping_GH100: Static bar1 mapped offset 0x0 size 0x5e9200000 Perfect, we now have the VRAM mapped. However, it\u0026rsquo;s not that easy to get P2P. When you run ./simpleP2P from cuda-samples, you get this error.\n1 2 3 4 5 6 7 8 [ 3742.840689] NVRM: kbusCreateP2PMappingForBar1P2P_GH100: added PCIe BAR1 P2P mapping between GPU2 and GPU3 [ 3742.840762] NVRM: kbusCreateP2PMappingForBar1P2P_GH100: added PCIe BAR1 P2P mapping between GPU3 and GPU2 [ 3742.841089] NVRM: nvAssertFailed: Assertion failed: (shifted \u0026gt;\u0026gt; pField-\u0026gt;shift) == value @ field_desc.h:272 [ 3742.841106] NVRM: nvAssertFailed: Assertion failed: (shifted \u0026amp; pField-\u0026gt;maskPos) == shifted @ field_desc.h:273 [ 3742.841281] NVRM: nvAssertFailed: Assertion failed: (shifted \u0026gt;\u0026gt; pField-\u0026gt;shift) == value @ field_desc.h:272 [ 3742.841292] NVRM: nvAssertFailed: Assertion failed: (shifted \u0026amp; pField-\u0026gt;maskPos) == shifted @ field_desc.h:273 [ 3742.865948] NVRM: GPU at PCI:0000:01:00: GPU-49c7a6c9-e3a8-3b48-f0ba-171520d77dd1 [ 3742.865956] NVRM: Xid (PCI:0000:01:00): 31, pid=21804, name=simpleP2P, Ch 00000013, intr 00000000. MMU Fault: ENGINE CE3 HUBCLIENT_CE1 faulted @ 0x7f97_94000000. Fault is of type FAULT_INFO_TYPE_UNSUPPORTED_KIND ACCESS_TYPE_VIRT_WRITE Failing with an MMU fault. So you dive into this and find that it\u0026rsquo;s using GMMU_APERTURE_PEER as the mapping type. That doesn\u0026rsquo;t seem supported in the 4090. So let\u0026rsquo;s see what types are supported, GMMU_APERTURE_VIDEO,GMMU_APERTURE_SYS_NONCOH, and GMMU_APERTURE_SYS_COH. We don\u0026rsquo;t care about being coherent with the CPU\u0026rsquo;s L2 cache, but it does have to go out the PCIe bus, so we rewrite GMMU_APERTURE_PEER to GMMU_APERTURE_SYS_NONCOH. We also no longer set the peer id that was corrupting the page table.\n1 2 3 4 5 6 7 cudaMemcpyPeer / cudaMemcpy between GPU0 and GPU1: 24.21GB/s Preparing host buffer and memcpy to GPU0... Run kernel on GPU1, taking source data from GPU0 and writing to GPU1... Run kernel on GPU0, taking source data from GPU1 and writing to GPU0... Copy data back to host from GPU0 and verify results... Verification error @ element 1: val = 0.000000, ref = 4.000000 Verification error @ element 2: val = 0.000000, ref = 8.000000 Progress! ./simpleP2P appears to work, however the copy isn\u0026rsquo;t happening. The address is likely wrong. It turns out they have a separate field for the peer address called fldAddrPeer, we change that to fldAddrSysmem. We also print out the addresses and note that the physical BAR address isn\u0026rsquo;t being added properly, they provide a field fabricBaseAddress for GMMU_APERTURE_PEER, we reuse it and put the BAR1 base address in there.\n","date":"2025-02-07T11:18:00+08:00","permalink":"https://charlie0129.github.io/blog/p/rtx4090-p2p-unlocked/","title":"RTX4090 GPUDirect P2P Unlocked!"},{"content":"Stripe Size When you create an hardware RAID array, you must encountered the term stripe size.\nFor example, an 8-disk RAID 1/0 has a stripe width of 4, with a stripe element size of 64 KB has a stripe size of 256 KB (4 * 64 KB). A 5-disk RAID 5 (4+1) with a 64 KB stripe element size also has a stripe width of 256 KB drive (4 * 64 KB) .\nA stripe is the smallest chunk of data within a RAID array that can be addressed. People often also refer to this as granularity or block size. It can be compared to the blocks (logical block addressing - LBA) on conventional hard drives. Most RAID controllers allow the user to define her or his favorite stripe size, because it alters the performance characteristics of a RAID array. Reference\nIt is important to let the filesystem you use know the underlying RAID array\u0026rsquo;s stripe size. This is because the filesystem will align its data blocks to the RAID array\u0026rsquo;s stripe size. This will help to avoid read-modify-write operations, which can be very slow.\nFor example, when creating a Virtual Disk in a Dell PERC H730P Controller, you can see the stripe element size.\nTune XFS for Hardware RAID When you format the XFS partition, you can tell XFS about the underlying array info.\nFor example, I have 4x8T HDDs in RAID 10 with stripe element size of 64 KB. So I have a stripe width of 2 and a stripe size of 128 KB. The corresponding XFS options is sw=2 and su=64k.\n1 mkfs.xfs -f -d su=64k,sw=2 /dev/sdb Other Resources https://www.percona.com/blog/aligning-io-on-a-hard-disk-raid-the-theory/ https://www.percona.com/blog/setting-up-xfs-the-simple-edition/ ","date":"2024-11-05T14:40:00+08:00","permalink":"https://charlie0129.github.io/blog/p/xfs-with-hw-raid/","title":"XFS Tuning for Hardware RAID"},{"content":"Background Consider a small lab, students need to use GPU for their projects. We have a NVIDIA GPU in our Proxmox VE server, and we want to share the GPU between multiple containers so that multiple students can use the GPU at the same time.\nWhy not use a VM? Because a GPU can only be passed through to one VM at a time (only one student can use the GPU at a time). And resources are not flexible in VMs.\nWhy not create multiple users in the host and let them run their programs in the host? Because we want to isolate the students from the host, so that they can\u0026rsquo;t access the host and other students\u0026rsquo; data.\nWhy not use Docker? Because Docker containers doesn\u0026rsquo;t have a full init system, and it\u0026rsquo;s hard to run some applications.\nSince we use PVE and it has LXC containers built-in (called CT), it is a perfect choice.\nInstall Drivers on the Host Make sure the GPU is detected by the host. Note the NVIDIA GPUs 3b:00.0 (Your address may differ).\n1 2 3 4 5 # lspci | grep -i nvidia 3b:00.0 VGA compatible controller: NVIDIA Corporation TU104GL [Quadro RTX 5000] (rev a1) 3b:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1) 3b:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1) 3b:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1) You may ask: does your entire lab only own one RTX 5000? What kind of lab is this? Are you cave people?\nYes, although we have multiple projects worth over millions of Chinese Yuan, most of the money is gone to the some other places (which I cannot publicly speak on the Internet 🤫 ). And the professors have no emphasis on students\u0026rsquo; growth. As a result, we are actually poor as hell.\nSince almost no one knows how to properly configure a Linux server, I want to help my classmates to learn more and let them use the only GPU. But to be honest, I won\u0026rsquo;t benefit from doing this. It\u0026rsquo;s just voluntary work.\nInstall prerequisites. Note that I am using pve-headers-$(uname -r) to install the headers for the current kernel. If you are using a different kernel, you may need to install the headers for that kernel. Also, you may want to use linux-headers-$(uname -r) instead of pve-headers-$(uname -r) if you are not using Proxmox VE.\n1 # apt install -y gcc make pve-headers-$(uname -r) Download CUDA toolkit from here and install it. Drivers are included in the CUDA toolkit so you don\u0026rsquo;t need to install drivers separately.\n1 2 # wget \u0026lt;cuda-runfile-download-url\u0026gt; # ./cuda_12.2.2_535.104.05_linux.run --silent The default installation options will work fine. If anything fails, you can check the log file at /var/log/cuda-installer.log for CUDA logs and /var/log/nvidia-installer.log for NVIDIA driver logs.\nPS: You need to blacklist nouveau driver. This is automatically done by PVE. If not, you can do this by creating a file /etc/modprobe.d/blacklist-nouveau.conf with the following content: blacklist nouveau. Then run update-initramfs -u to update the initramfs.\nPPS: If you used to passthrough this GPU to a VM, be sure to remove the GPU from the VM\u0026rsquo;s hardware configuration in PVE otherwise PVE will bound the GPU to vfio-pci (see Kernel driver in use row in lspci -k) and cannot be used by the host.\nPPPS: Some kernel versions are known to have problems with NVIDIA drivers. If you encounter problems, you may need to downgrade/upgrade the kernel. For example, kernel version 5.10.0 is known to have make[3]: *** No rule to make target 'scripts/module.lds', needed by '/tmp/selfgz38416/NVIDIA-Linux-x86_64-560.35.03/kernel-open/nvidia.ko' error.\nAfter installation finished, check if the driver is loaded.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 # nvidia-smi Tue Nov 5 09:56:44 2024 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 Quadro RTX 5000 Off | 00000000:3B:00.0 Off | Off | | 33% 44C P0 28W / 230W | 0MiB / 16384MiB | 6% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | No running processes found | +---------------------------------------------------------------------------------------+ Allow NVIDIA Device Passthrough in CT Now we need to allow the CT to access the GPU. I am using an unprivileged container here. Edit the CT\u0026rsquo;s configuration file (/etc/pve/local/lxc/\u0026lt;id\u0026gt;.conf). Add the following lines to the end of the file.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 arch: amd64 cores: 4 features: nesting=1 hostname: ct-gpu-tmpl-deb127-cu122 memory: 4096 net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=AA:AB:F0:07:42:D0,ip=dhcp,type=veth ostype: debian rootfs: local-zfs:basevol-8001-disk-0,size=16G swap: 0 unprivileged: 1 # These lines allow the container to access specific character devices (c) with rwm # permissions (read, write, modify). These are needed for NVIDIA GPU access. + lxc.cgroup.devices.allow: c 195:* rwm + lxc.cgroup.devices.allow: c 509:* rwm + lxc.cgroup.devices.allow: c 235:* rwm # These lines mount various GPU-related devices from the host into the container. + lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file + lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file + lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file + lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file + lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file + lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir + lxc.mount.entry: /dev/fb0 dev/fb0 none bind,optional,create=file PS: If you cannot use nvidia-smi (it says Failed to initialize NVML: Unknown Error), there is a possibility that you are using cgroup2. Change all the lxc.cgroup.devices.allow lines to lxc.cgroup2.devices.allow.\nExplanation:\nAllows container access to NVIDIA device nodes:\nc 195:* - NVIDIA character devices c 509:* - NVIDIA UVM devices c 235:* - NVIDIA CTL devices Maps the following host GPU devices into container:\n/dev/nvidia0 - Main GPU device /dev/nvidiactl - NVIDIA control device /dev/nvidia-modeset - Display mode setting /dev/nvidia-uvm - Unified memory management /dev/nvidia-uvm-tools - UVM diagnostic tools /dev/dri - Direct Rendering Infrastructure /dev/fb0 - Framebuffer device Mount options:\nbind: Mount as a bind mount optional: Don\u0026rsquo;t fail if device doesn\u0026rsquo;t exist create=file/dir: Create the mount point if it doesn\u0026rsquo;t exist Note that if you are using a different GPU, you may need to change the device numbers. For example, /dev/nvidia1 instead of /dev/nvidia0. You can find the device numbers in nvidia-smi output.\nInstall Drivers in CT Log into the CT. All the following commands are run in the CT.\nYou should be able to see NVIDIA devices inside the CT:\n1 2 3 4 5 6 # ls -l /dev/nvidia* ---------- 1 root root 0 Nov 5 02:31 /dev/nvidia-modeset crw-rw-rw- 1 nobody nogroup 507, 0 Nov 5 01:56 /dev/nvidia-uvm crw-rw-rw- 1 nobody nogroup 507, 1 Nov 5 01:56 /dev/nvidia-uvm-tools crw-rw-rw- 1 nobody nogroup 195, 0 Nov 5 01:56 /dev/nvidia0 crw-rw-rw- 1 nobody nogroup 195, 255 Nov 5 01:56 /dev/nvidiactl Install CUDA and drivers, just like you would on a physical machine, except that you don\u0026rsquo;t need to install the kernel modules. I will install CUDA 12.2 (drivers are included in the CUDA installer).\n1 2 3 # wget \u0026lt;cuda-runfile-download-url\u0026gt; # apt install -y gcc # ./cuda_12.2.2_535.104.05_linux.run --extract=$(pwd)/cu122 Note that I extracted the installer to manually install it because we want to skip kernel module installation and such options are not exposed in the installer.\nInstall the bundled drivers:\n1 2 cd cu122 ./NVIDIA-Linux-x86_64-535.104.05.run --no-nouveau-check --no-kernel-modules --silent Run nvidia-smi to check if the driver is loaded.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 # nvidia-smi Tue Nov 5 05:49:02 2024 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 Quadro RTX 5000 Off | 00000000:3B:00.0 Off | Off | | 33% 38C P0 23W / 230W | 0MiB / 16384MiB | 0% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | No running processes found | +---------------------------------------------------------------------------------------+ We can now see that the GPU is accessible in the CT.\nLet\u0026rsquo;s continue with the CUDA installation. Remember to uncheck the driver installation option because we have already installed the drivers above.\n1 ./cuda-linux.12.2.2-535.104.05.run After a successful installation, you should add cuda binaries to PATH. Instructions should be printed at the end of the installation. Then you can run nvcc to see if CUDA is installed correctly.\nEverything should be working by this point.\nMissing nvidia-uvm and High Idle Power Draw One problem I encountered is that when the host reboots, the GPU is not accessible in the CT. This is because nvidia-uvm device isn\u0026rsquo;t created until an application attempts to interact with the graphics card. This is a problem because no application will interact with the GPU at boot, so no nvidia-uvm device is created. But the CT needs the nvidia-uvm device bind-mounted at CT-startup in order to access the GPU.\nAlso, the graphics card have insanely high power draw at idle (over 100 Watts). The GPU is in P0 and never leaves it. We can use nvidia-persistenced to let the GPU enter a low-power state (P8) when not in use.\nTo solve this, we can run nvidia-smi -pm 1 (which enables nvidia-persistenced and keeps nvidia character device and handles frequency scaling) at boot. Add the following line to the host\u0026rsquo;s crontab to run nvidia-smi -pm 1 at boot.\nPS: This only works if the host is a headless server (no monitor attached). If you have a monitor attached, you may need to run nvidia-smi below instead.\n1 2 # crontab -e @reboot /usr/bin/nvidia-smi -pm 1 Downsides Despite the fact that this method works best for us, there are some downsides:\nThe CT will have full access to the GPU. If one CT uses all the GPU memory, other CTs will be starving. So you must trust the users of the CTs. This is not a problem for us because we know each other. Driver updates are a bit more complicated. You need to update the drivers on the host and in all of the CTs. It\u0026rsquo;s best to not update the drivers too often. The CTs share the same kernel with the host. To avoid potential compatibility issues, we don\u0026rsquo;t update the kernel unless necessary. ","date":"2024-11-04T12:57:00+08:00","permalink":"https://charlie0129.github.io/blog/p/pve-ct-share-nvidia-gpu/","title":"Share NVIDIA GPU between CTs in Proxmox VE"},{"content":"Background One particular issue that I encountered when I tried to install some software using Homebrew on my Mac is that Homebrew deprecates support for older macOS versions just like Apple do. Once a macOS version is deprecated, Homebrew will not provide bottles (precompiled binaries) for that version. This means that you have to compile the software from source, which is extremely time-consuming and sometimes error-prone. Imagine installing shell-check will require to build ghc from source, which takes hours, and then use ghc to build shell-check from source. This is not a good experience.\nSolution Pin homebrew/core to an older version One way to work around this issue is to use a older version of the homebrew/core tap, such that the bottles are still available for the macOS version you are using because they are built when your macOS version is supported. The downside is that you will be using outdated software. If this is not an issue for you then congratulations. You can achieve this by running the following command:\n1 2 3 $ brew tap homebrew/core $ cd $(brew --repo homebrew/core) $ git reset --hard \u0026lt;commit-hash\u0026gt; Make sure you disable the auto-update of Homebrew so that it does not update the homebrew/core tap to the latest version. You can do this by running the following command:\n1 $ echo \u0026#39;export HOMEBREW_NO_AUTO_UPDATE=1\u0026#39; \u0026gt;\u0026gt; ~/.zshrc Fake macOS version Another way is to fake the macOS version that Homebrew detects. This way, Homebrew will provide bottles for the macOS version you are faking. Most software will work and you will be using the latest software. However, this may not work for all software as some software may have dependencies that are not available for the macOS version you are faking. You can achieve this by running the following command:\n1 2 $ export HOMEBREW_FAKE_MACOS=13.0 # Choose a macOS version that is supported by Homebrew and close to your macOS version $ brew install xxx Explanation Homebrew actually has an private undocumented API that allows you to fake the macOS version. https://github.com/Homebrew/brew/blob/a3d8f4e0e4a22da9990d59cca70bec1e7be726cf/Library/Homebrew/os/mac.rb#L41\n1 2 3 4 5 6 7 8 9 10 11 12 # This can be compared to numerics, strings, or symbols # using the standard Ruby Comparable methods. # # @api internal sig { returns(MacOSVersion) } def self.full_version @full_version ||= if (fake_macos = ENV.fetch(\u0026#34;HOMEBREW_FAKE_MACOS\u0026#34;, nil)) # for Portable Ruby building MacOSVersion.new(fake_macos) else MacOSVersion.new(VERSION) end end Example I have macOS Monterey (12.0) running, which is just deprecated by Homebrew. I want to install batt which is not available as a bottle for macOS Monterey.\nIf I run brew install batt, I will get the following error:\n1 2 3 4 5 6 7 8 9 10 11 12 $ brew install batt Warning: You are using macOS 12. We (and Apple) do not provide support for this old version. It is expected behaviour that some formulae will fail to build in this old version. It is expected behaviour that Homebrew will be buggy and slow. Do not create any issues about this on Homebrew\u0026#39;s GitHub repositories. Do not create any issues even if you think this message is unrelated. Any opened issues will be immediately closed without response. Do not ask for help from Homebrew or its maintainers on social media. You may ask for help in Homebrew\u0026#39;s discussions but are unlikely to receive a response. Try to figure out the problem yourself and submit a fix as a pull request. We will review it but may or may not accept it. It\u0026rsquo;s Oct 8, 2024 and the latest macOS version is macOS Sequoia (15.0). So the last 3 supported macOS versions are:\nmacOS Sequoia (15.0) macOS Sonoma (14.0) macOS Ventura (13.0) You can confirm that by checking out batt\u0026rsquo;s formula:\n1 2 3 4 5 6 $ brew edit batt ... sha256 cellar: :any_skip_relocation, arm64_sequoia: \u0026#34;61bd7790a82f2269b9a0ce1585c57564d03be4e4ac89b8f0d8843c0073a688e6\u0026#34; sha256 cellar: :any_skip_relocation, arm64_sonoma: \u0026#34;198bd7bb9a808f0a9e4cb1a31b7e9c2a72d690ba1d07ebddf78b6d0ce0b6dd03\u0026#34; sha256 cellar: :any_skip_relocation, arm64_ventura: \u0026#34;6eff598159b263327b8b562ab32f5e5e7157c20f25cbecfa08b48eda794c4c43\u0026#34; ... Showing only the bottles for macOS Sequoia (15.0), macOS Sonoma (14.0), and macOS Ventura (13.0) are available.\nSadly my macOS version (Monterey 12.0) is not in the list, but macOS Ventura (13.0) is the closest to macOS Monterey (12.0). So I can fake the macOS version to macOS Ventura (13.0) by running the following command:\n1 2 3 4 5 6 7 8 9 10 11 $ export HOMEBREW_FAKE_MACOS=13.0 $ brew config ... macOS: 13.0-arm64 # YES! I\u0026#39;m now faking macOS Ventura (13.0) ... $ brew install batt ==\u0026gt; Downloading https://ghcr.io/v2/homebrew/core/batt/manifests/0.3.1 Already downloaded: /Users/charlie/Library/Caches/Homebrew/downloads/86da7d77f0bacb44475e0280eef162148e3b51df8ea44ff9bc16a5a7bba6d39f--batt-0.3.1.bottle_manifest.json ==\u0026gt; Fetching batt ==\u0026gt; Downloading https://ghcr.io/v2/homebrew/core/batt/blobs/sha256:6eff598159b263327b8b562ab32f5e5e7157c20f25cbecfa08b48eda794c4c43 Already downloaded: /Users/charlie/Library/Caches/Homebrew/downloads/24fc002de2ee11096a544dd84ff1033262a5cb627c2b824c915a552bb6a588dd--batt--0.3.1.arm64_ventura.bottle.tar.gz As you can see, Homebrew is downloading the bottle for macOS Ventura (13.0) successfully! (Of course, the log is showing that I have already downloaded earlier, but you get the idea.)\nbatt itself does not rely on any APIs that are not available in my current macOS version, so it works perfectly fine.\n","date":"2024-10-08T22:08:00+08:00","permalink":"https://charlie0129.github.io/blog/p/homebrew-fake-macos-version/","title":"Fake macOS Version Detected by Homebrew"},{"content":"Background We want to test some sandbox application on our new cluster. My colleague reported that the performance is not as expected and wanted me to take a look.\nThe sandbox application works like this: it starts a python container, runs some scripts, and exits. They executes scripts provided by the user, so the cluster is expected to create and destroy a lot of containers in a short period of time.\nAssumptions Considering the characteristics of the application, the bottleneck could be the following:\nCPU/Memory of the worker nodes, or the container runtime Performance of the master node (apiserver, etcd, and etc) Since we have relatively beefy worker nodes, each with 2*Xeon Platinum 8353v (72C144T) and 512GB of memory, the bottleneck is likely the master node (8C 16GB memory).\nLet\u0026rsquo;s verify this assumption.\nObservations I ran k6 to load test the cluster.\nPods started to pending\u0026hellip;\nHmm, it smelled like a etcd issue (the newly-created Pods is not able to be written to the database). I looked at the etcd dashboard. Sure enough, db backend is having some trouble writing the files.\nI grabed the the disk stats: the disks on the master node is almost fully utilized! A better disk (SSD) is really needed.\nSide note We also experienced unexpected issue when I applied a cilium config, the whole cluster goes down (Thanks god. This is not a production cluster, or this will be a total disaster). At first, I assumed the cilium config that I written is wrong and I reverted, hoping the cluster will be fixed. Nope, all nodes are down. However, some interesting behavior is noticed: all worker nodes are in a ready - notready loop.\nUpon further investigation, I noticed the apiserver is constantly being OOM killed.\nThe apiserver is using more memory than the node has.\nIt seems something is hitting the apiserver really hard. Under normal circumstances, I will go to the master node to look at the apiserver logs. However, the master node is down (because OOM) and I cannot ssh into it.\nThe cluster is stuck in such a loop:\nWorker nodes are all starting (cilium starting) and pulling/changed config through apiserver apiserver on the master node OOM\u0026rsquo;ed because of bursts of requests the apiserver is OOM killed worker node cannot report status to the apiserver worker node is marked as not ready; cilium pods are terminated because of nodes are not ready go to step 1 Since the cluster cannot properly start in such a scenario, I have to use some dirty fix to start the cluster: just after the apiserver is started, and before the apiserver eats all memory, such that the master node is still alive, quickly SSH into the master node and kill the apiserver process. By doing this a few times, we can span the burst of requests over a longer period of time, avoiding overloading the apiserver (causing it to be killed).\nSolution The solution is simple: replace the master nodes with faster ones (much more memory, NVME SSDs).\n","date":"2024-08-15T14:54:00+08:00","permalink":"https://charlie0129.github.io/blog/p/what-happens-when-k8s-master-have-low-memory-or-slow-disk/","title":"What happens when your Kubernetes master has low memory or slow disk?"},{"content":"Background The RDMA NIC names (InfiniBand HCAs in this case) of our nodes are inconsistent. We want to rename them to a consistent naming scheme.\nOne node:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 root@tj01-h20-node139:/# ibstatus Infiniband device \u0026#39;ibp171s0\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x70 sm lid:\t0x4 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t200 Gb/sec (2X NDR) link_layer:\tInfiniBand Infiniband device \u0026#39;ibp187s0\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x6f sm lid:\t0x4 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t200 Gb/sec (2X NDR) link_layer:\tInfiniBand Infiniband device \u0026#39;ibp203s0\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x29 sm lid:\t0x4 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t200 Gb/sec (2X NDR) link_layer:\tInfiniBand Infiniband device \u0026#39;ibp219s0\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x6a sm lid:\t0x4 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t200 Gb/sec (2X NDR) link_layer:\tInfiniBand Infiniband device \u0026#39;ibp41s0\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x6c sm lid:\t0x4 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t200 Gb/sec (2X NDR) link_layer:\tInfiniBand Infiniband device \u0026#39;ibp59s0\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x2b sm lid:\t0x4 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t200 Gb/sec (2X NDR) link_layer:\tInfiniBand Infiniband device \u0026#39;ibp75s0\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x80 sm lid:\t0x4 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t200 Gb/sec (2X NDR) link_layer:\tInfiniBand Infiniband device \u0026#39;ibp93s0\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x6e sm lid:\t0x4 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t200 Gb/sec (2X NDR) link_layer:\tInfiniBand Infiniband device \u0026#39;rocep22s0f0\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x0 sm lid:\t0x0 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t25 Gb/sec (1X EDR) link_layer:\tEthernet Another node:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 root@tj01-h20-node140:/# ibstatus Infiniband device \u0026#39;mlx5_2\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x68 sm lid:\t0x4 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t200 Gb/sec (2X NDR) link_layer:\tInfiniBand Infiniband device \u0026#39;mlx5_3\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x71 sm lid:\t0x4 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t200 Gb/sec (2X NDR) link_layer:\tInfiniBand Infiniband device \u0026#39;mlx5_4\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x27 sm lid:\t0x4 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t200 Gb/sec (2X NDR) link_layer:\tInfiniBand Infiniband device \u0026#39;mlx5_5\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x69 sm lid:\t0x4 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t200 Gb/sec (2X NDR) link_layer:\tInfiniBand Infiniband device \u0026#39;mlx5_6\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x6d sm lid:\t0x4 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t200 Gb/sec (2X NDR) link_layer:\tInfiniBand Infiniband device \u0026#39;mlx5_7\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x6b sm lid:\t0x4 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t200 Gb/sec (2X NDR) link_layer:\tInfiniBand Infiniband device \u0026#39;mlx5_8\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x2a sm lid:\t0x4 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t200 Gb/sec (2X NDR) link_layer:\tInfiniBand Infiniband device \u0026#39;mlx5_9\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x2c sm lid:\t0x4 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t200 Gb/sec (2X NDR) link_layer:\tInfiniBand Infiniband device \u0026#39;mlx5_bond_0\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x0 sm lid:\t0x0 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t25 Gb/sec (1X EDR) link_layer:\tEthernet Note that one node has the names ibp* and the other has mlx5_*. We want to make node 1 follow a same naming scheme as the other nodes.\nTo be fair ibp* is actually consistent (named using PCI location). mlx5_* is not consistent, which depends on the card initialization order and can change after a reboot.\nAnyway, since all other nodes are using mlx5_* naming scheme, we want to rename all of them to mlx5_* for consistency with the other nodes.\nSolution 1 cp cp /lib/udev/rules.d/60-rdma-persistent-naming.rules /etc/udev/rules.d/ This is all you need. Now reboot your server.\nIf you are interested in the details, read on.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 # SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB) # Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file # # Rename modes: # NAME_FALLBACK - Try to name devices in the following order: # by-pci -\u0026gt; by-guid -\u0026gt; kernel # NAME_KERNEL - leave name as kernel provided # NAME_PCI - based on PCI/slot/function location # NAME_GUID - based on system image GUID # NAME_FIXED - rename the device to the fixed named in the next argument # # The stable names are combination of device type technology and rename mode. # Infiniband - ib* # RoCE - roce* # iWARP - iw* # OPA - opa* # Default (unknown protocol) - rdma* # # Example: # * NAME_PCI # pci = 0000:00:0c.4 # Device type = IB # mlx5_0 -\u0026gt; ibp0s12f4 # * NAME_GUID # GUID = 5254:00c0:fe12:3455 # Device type = RoCE # mlx5_0 -\u0026gt; rocex525400c0fe123455 # ACTION==\u0026#34;add\u0026#34;, SUBSYSTEM==\u0026#34;infiniband\u0026#34;, PROGRAM=\u0026#34;rdma_rename %k NAME_KERNEL\u0026#34; # Example: # * NAME_FIXED # fixed name for specific board_id # #ACTION==\u0026#34;add\u0026#34;, ATTR{board_id}==\u0026#34;MSF0010110035\u0026#34;, SUBSYSTEM==\u0026#34;infiniband\u0026#34;, PROGRAM=\u0026#34;rdma_rename %k NAME_FIXED myib\u0026#34; Look at the line contains PROGRAM=\u0026quot;rdma_rename %k NAME_KERNEL\u0026quot;. So if you want to use mlx5_* (called kernel names), you can use NAME_KERNEL as the rename mode (default). If you want to use ibp* (called PCI names), you can use NAME_PCI as the rename mode or use NAME_FALLBACK, which first tries NAME_PCI.\n","date":"2024-08-09T14:28:00+08:00","permalink":"https://charlie0129.github.io/blog/p/rename-rnics/","title":"Rename RDMA NIC Interface Names"},{"content":"So I decided to test NFS over RDMA somehow.\nHostname ft2000 (Server) t3640 (Client) CPU Phytium FT-2000+/64 Intel Core i9-10900K RAM Quad Channel DDR4 128GB Dual Channel DDR4 64GB NIC Mellanox ConnextX-4 Lx 25GbE Mellanox ConnextX-4 Lx 25GbE OS UOS Server 20 1070a Debian 12 Two servers are connected using a 25GbE network. The goal is to set up NFS over RDMA and benchmark it to see the performance difference between NFS over TCP and NFS over RDMA.\nRDMA Link Verification Install RDMA/IB dependencies:\nDependencies should be the same on both devices (no server and client differences). The difference is only the package manager (and OS).\nIf you are using Mellanox OFED, you can skip this step. Be sure to install Mellanox OFED using --with-nfsrdma flag. Otherwise, you will not be able to use NFS over RDMA.\nOn UOS Server 20 1070a, which is Anolis8-based, which is again CentOS8-based:\n1 2 3 4 root@ft2000:/# yum install -y libibverbs librdmacm opensm-libs rdma-core rdma-core-devel \\ librdmacm-utils opensm-static srp_daemon ucx-devel ucx-rdmacm infiniband-diags ibacm \\ opensm-devel ucx ucx-ib libibumad opensm mstflint ucx-cma openmpi rshim libibverbs-utils \\ perftest rdma-core rdma-core-devel infiniband-diags rshim On Debian 12:\n1 2 3 4 root@t3640:/# apt install -y infiniband-diags srptools perftest opensm-doc librdmacm-dev \\ rdmacm-utils librdmacm1 ibacm libibmad-dev libibmad5 libibumad-dev libibumad3 \\ ibverbs-utils libibverbs-dev libibverbs1 mstflint rdma-core opensm fio librbd1 \\ librados2 libibnetdisc5 ibverbs-providers Make sure link is up on both servers using ip or ibstatus command:\nClient:\nUse ip (note state UP):\n1 2 root@t3640:/# ip l show dev enp1s0f1np1 6: enp1s0f1np1: \u0026lt;BROADCAST,MULTICAST,UP,LOWER_UP\u0026gt; mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 Or use ibstatis (note phys state: 5: LinkUp):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 root@t3640:/# ibstatus Infiniband device \u0026#39;mlx5_0\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x0 sm lid:\t0x0 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t10 Gb/sec (1X QDR) link_layer:\tEthernet Infiniband device \u0026#39;mlx5_1\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x0 sm lid:\t0x0 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t25 Gb/sec (1X EDR) link_layer:\tEthernet Server:\nUse ip (note state UP):\n1 2 root@ft2000:/# ip l show dev enp16s0f0np0 4: enp16s0f0np0: \u0026lt;BROADCAST,MULTICAST,UP,LOWER_UP\u0026gt; mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 Or use ibstatis (note phys state: 5: LinkUp):\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 root@ft2000:/# ibstatus Infiniband device \u0026#39;mlx5_0\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x0 sm lid:\t0x0 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t25 Gb/sec (1X EDR) link_layer:\tEthernet Infiniband device \u0026#39;mlx5_1\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x0 sm lid:\t0x0 state:\t1: DOWN phys state:\t3: Disabled rate:\t40 Gb/sec (4X QDR) link_layer:\tEthernet Verify IP connectivity between the servers (ping the IP address of the other server):\n1 2 3 4 root@t3640:/# ping 192.168.23.1 PING 192.168.23.1 (192.168.23.1) 56(84) bytes of data. 64 bytes from 192.168.23.1: icmp_seq=1 ttl=64 time=0.199 ms 64 bytes from 192.168.23.1: icmp_seq=2 ttl=64 time=0.206 ms Make sure the the InfiniBand kernel modules are enabled.\n1 2 3 4 5 6 7 8 9 10 # On both servers root@ft2000:/# lsmod | grep ^ib ib_srpt 262144 0 ib_isert 262144 0 ib_iser 262144 0 ib_umad 262144 0 ib_ipoib 327680 0 ib_cm 262144 3 rdma_cm,ib_ipoib,ib_srpt ib_uverbs 327680 2 rdma_ucm,mlx5_ib ib_core 458752 12 rdma_cm,ib_ipoib,rpcrdma,ib_srpt,iw_cm,ib_iser,ib_umad,ib_isert,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm Make sure you have a lossless network:\nIn case the RDMA is running over Ethernet (RoCE) you need to make sure that the network is configured to be loss-less, which means that either flow control (FC) or priority flow control PFC is enabled on the adapter ports and the switch.\nIn case of lab environment or small setup, you can use Global Pause Flow Control to create loss-less environment. To check what is the global pause configuration use the following command (by default it is enabled normally).\n1 2 3 4 5 root@t3640:/# ethtool -a enp1s0f0np0 Pause parameters for enp1s0f0np0: Autonegotiate:\toff RX:\ton TX:\ton In case it is disabled, run:\n1 root@t3640:/# ethtool -A enp1s0f0np0 rx on tx on Test RDMA speed. Refer to my previous blog InfiniBand Performance Test for details.\nRun the following command on client and server, respectively:\n1 2 root@t3640:/# ib_send_bw --report_gbit -F -d mlx5_1 # device name can be seen in `ibstatus` command 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 root@ft2000:/# ib_send_bw --report_gbit -a -F -d mlx5_0 192.168.23.2 --------------------------------------------------------------------------------------- Send BW Test Dual-port : OFF\tDevice : mlx5_0 Number of qps : 1\tTransport type : IB Connection type : RC\tUsing SRQ : OFF PCIe relax order: ON ibv_wr* API : ON TX depth : 128 CQ Moderation : 100 Mtu : 1024[B] Link type : Ethernet GID index : 3 Max inline data : 0[B] rdma_cm QPs\t: OFF Data ex. method : Ethernet --------------------------------------------------------------------------------------- local address: LID 0000 QPN 0x0087 PSN 0x176d37 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:23:01 remote address: LID 0000 QPN 0x0188 PSN 0x789e84 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:23:02 --------------------------------------------------------------------------------------- #bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps] 65536 1000 23.14 23.13 0.044119 --------------------------------------------------------------------------------------- As you can see, the speed is around 23 Gb/sec. This is the maximum speed of the network (25G). If you see a lower speed, you need to check the network configuration and the switch configuration.\nConfigure NFS without RDMA Just to make sure NFS is working.\nInstall NFS on server:\n1 2 3 root@ft2000:/# yum -y install nfs-utils # On Debian: apt install nfs-kernel-server root@ft2000:/# systemctl start nfs-server rpcbind root@ft2000:/# systemctl enable nfs-server rpcbind Use /tmp as a test dir and share it:\nSince I want to test the network performance (TCP and RDMA), I don\u0026rsquo;t want to be bottlenecked by the disk speed. So I will use /tmp, which is backed by RAM (usually), as a reasonably fast directory to avoid bottlenecks.\nMake sure it is tmpfs, otherwise IO will be limited by your disk speed.\n1 2 root@ft2000:/# mount | grep /tmp tmpfs on /tmp type tmpfs (rw,nosuid,nodev) 1 root@ft2000:/# echo \u0026#39;/tmp *(rw,async,insecure,no_root_squash)\u0026#39; \u0026gt;\u0026gt;/etc/exports Make it available:\n1 2 root@ft2000:/# exportfs -a root@ft2000:/# systemctl restart nfs-server Install NFS on client:\n1 root@t3640:/# apt install nfs-common Mount the NFS share temporarily:\n1 2 root@t3640:/# mkdir -p /mnt/nfs root@t3640:/# mount -t nfs 192.168.23.1:/tmp /mnt/nfs Verify the mount:\nShould see the files in /tmp on the server.\n1 root@t3640:/# ls -l /mnt/nfs Configure NFS with RDMA Load RDMA transport module on the server:\n1 2 3 root@ft2000:/# modprobe svcrdma # To make it permanent: echo \u0026#39;svcrdma\u0026#39; \u0026gt;\u0026gt;/etc/modules root@ft2000:/# systemctl restart nfs-server Instruct the server to listen on the RDMA transport port (20049 is the default port):\nNote: if you see echo: write error: protocol not supported, it means that the NFSoRDMA is not supported in the Mellanox OFED. You need to use the inbox OS driver (refer to Install RDMA/IB dependencies section in this blog) or re-install Mellanox OFED with --with-nfsrdma flag. I was tripped by this issue.\n1 2 root@ft2000:/# echo rdma 20049 \u0026gt; /proc/fs/nfsd/portlist # To make it permanent: echo \u0026#39;rdma=nfsrdma\u0026#39; \u0026gt;\u0026gt; /etc/nfs.conf Load the RDMA transport module on the client\n1 2 3 root@t3640:/# modprobe xprtrdma # To make it permanent: echo \u0026#39;xprtrdma\u0026#39; \u0026gt;\u0026gt;/etc/modules root@t3640:/# systemctl restart nfs-utils.service Mount the NFS share with RDMA:\nBefore running the mount command, unmount the NFS share if it is mounted (umount /mnt/nfs).\n1 2 root@t3640:/# mount -t nfs -o rdma,port=20049 192.168.23.1:/tmp /mnt/nfs # To make it permanent: echo \u0026#39;192.168.23.1:/tmp /mnt/nfs nfs defaults,proto=rdma,port=20049 0 0\u0026#39; \u0026gt;\u0026gt; /etc/fstab Check the mount parameters:\nIf you see proto=rdma, it means the NFS is mounted using RDMA.\n1 2 3 root@t3640:/# nfsstat -m /mnt/nfs from 192.168.23.1:/tmp Flags:\trw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.23.2,local_lock=none,addr=192.168.23.1 Verify the mount:\nShould see the files in /tmp on the server.\n1 root@t3640:/# ls -l /mnt/nfs Benchmark Use fio to benchmark the NFS performance. Parameters such as bs, iodepth will be changed to different values later to evaluate the performance in different scenarios.\nI will only test random read performance since sequential read performance is not a main concern even using TCP let alone RDMA.\nThe actual command will be:\n1 2 3 root@t3640:/# fio --rw=randread --bs=4k --numjobs=1 --iodepth=1 --runtime=10 --time_based --loops=1 \\ --ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap \\ --exitall --name task1 --filename=/mnt/nfs/testfile --size=256M The full log and related scripts can be found in the Appendix.\nHere are the results we have all been waiting for!\nFor IO operations per second (IOPS), the average IOPS for NFS over RDMA is higher than NFS over TCP. The difference is more significant when the block size is larger and the most significant when iodepth=12. We can also see that no matter the block size and iodepth, the IOPS for NFS over TCP is capped at around 47 KIOPS, while the one of RDMA is significantly higher at over 180 KIOPS.\nThe reason why the difference is smaller with larger block sizes is that the network bandwidth is the bottleneck. For our 25GbE network, the maximum bandwidth is around 2800 MiB/s. We are hitting this limit with larger block sizes and this much IOPS. This is unrelated to the RDMA or TCP. If you have a faster network, you will see this difference for larger block sizes as well.\nFor 4K block size:\nIO Depth KIOPS (RDMA) KIOPS (TCP) Ratio 8 124.5 36.1 3.4 12 156.0 38.1 4.1 16 165.8 41.4 4.0 24 177.0 45.8 3.9 32 179.4 50.7 3.5 For bandwidth, the average bandwidth for NFS over RDMA is significantly higher (around 4x for small block sizes).\nThe reason why the difference is more significant when the block size is small (4-16) is that we are limited by the network bandwidth. Since we have a 25GbE network, the maximum bandwidth is around 2800 MiB/s so you can see that the bandwidth is capped at 2800 MiB/s for large block sizes and deep IO depths. If you have a faster network, you will see this 4x difference for larger block sizes as well.\nLatency-wise, the average latency for NFS over RDMA is slightly lower than NFS over TCP. The difference is more significant for small block size and large iodepths.\nAppendix Benchmark script benchmark.sh: automatically run fio with different bs and iodepth values and saves logs (it saves stdout, not the fio-generated log, e.g. --bandwidth-log). Plotting script plot.py: parses the fio logs (stdout) saved by benchmark.sh and plots the results, which is the three figures you see above in the Benchmark section. Raw fio Logs (Click to Show) rdma_fio_bs4k_iodepth1.log rdma_fio_bs4k_iodepth2.log rdma_fio_bs4k_iodepth3.log rdma_fio_bs4k_iodepth4.log rdma_fio_bs4k_iodepth6.log rdma_fio_bs4k_iodepth8.log rdma_fio_bs4k_iodepth12.log rdma_fio_bs4k_iodepth16.log rdma_fio_bs4k_iodepth24.log rdma_fio_bs4k_iodepth32.log rdma_fio_bs4k_iodepth64.log rdma_fio_bs4k_iodepth128.log rdma_fio_bs8k_iodepth1.log rdma_fio_bs8k_iodepth2.log rdma_fio_bs8k_iodepth3.log rdma_fio_bs8k_iodepth4.log rdma_fio_bs8k_iodepth6.log rdma_fio_bs8k_iodepth8.log rdma_fio_bs8k_iodepth12.log rdma_fio_bs8k_iodepth16.log rdma_fio_bs8k_iodepth24.log rdma_fio_bs8k_iodepth32.log rdma_fio_bs8k_iodepth64.log rdma_fio_bs8k_iodepth128.log rdma_fio_bs16k_iodepth1.log rdma_fio_bs16k_iodepth2.log rdma_fio_bs16k_iodepth3.log rdma_fio_bs16k_iodepth4.log rdma_fio_bs16k_iodepth6.log rdma_fio_bs16k_iodepth8.log rdma_fio_bs16k_iodepth12.log rdma_fio_bs16k_iodepth16.log rdma_fio_bs16k_iodepth24.log rdma_fio_bs16k_iodepth32.log rdma_fio_bs16k_iodepth64.log rdma_fio_bs16k_iodepth128.log rdma_fio_bs32k_iodepth1.log rdma_fio_bs32k_iodepth2.log rdma_fio_bs32k_iodepth3.log rdma_fio_bs32k_iodepth4.log rdma_fio_bs32k_iodepth6.log rdma_fio_bs32k_iodepth8.log rdma_fio_bs32k_iodepth12.log rdma_fio_bs32k_iodepth16.log rdma_fio_bs32k_iodepth24.log rdma_fio_bs32k_iodepth32.log rdma_fio_bs32k_iodepth64.log rdma_fio_bs32k_iodepth128.log rdma_fio_bs64k_iodepth1.log rdma_fio_bs64k_iodepth2.log rdma_fio_bs64k_iodepth3.log rdma_fio_bs64k_iodepth4.log rdma_fio_bs64k_iodepth6.log rdma_fio_bs64k_iodepth8.log rdma_fio_bs64k_iodepth12.log rdma_fio_bs64k_iodepth16.log rdma_fio_bs64k_iodepth24.log rdma_fio_bs64k_iodepth32.log rdma_fio_bs64k_iodepth64.log rdma_fio_bs64k_iodepth128.log rdma_fio_bs128k_iodepth1.log rdma_fio_bs128k_iodepth2.log rdma_fio_bs128k_iodepth3.log rdma_fio_bs128k_iodepth4.log rdma_fio_bs128k_iodepth6.log rdma_fio_bs128k_iodepth8.log rdma_fio_bs128k_iodepth12.log rdma_fio_bs128k_iodepth16.log rdma_fio_bs128k_iodepth24.log rdma_fio_bs128k_iodepth32.log rdma_fio_bs128k_iodepth64.log rdma_fio_bs128k_iodepth128.log rdma_fio_bs256k_iodepth1.log rdma_fio_bs256k_iodepth2.log rdma_fio_bs256k_iodepth3.log rdma_fio_bs256k_iodepth4.log rdma_fio_bs256k_iodepth6.log rdma_fio_bs256k_iodepth8.log rdma_fio_bs256k_iodepth12.log rdma_fio_bs256k_iodepth16.log rdma_fio_bs256k_iodepth24.log rdma_fio_bs256k_iodepth32.log rdma_fio_bs256k_iodepth64.log rdma_fio_bs256k_iodepth128.log rdma_fio_bs512k_iodepth1.log rdma_fio_bs512k_iodepth2.log rdma_fio_bs512k_iodepth3.log rdma_fio_bs512k_iodepth4.log rdma_fio_bs512k_iodepth6.log rdma_fio_bs512k_iodepth8.log rdma_fio_bs512k_iodepth12.log rdma_fio_bs512k_iodepth16.log rdma_fio_bs512k_iodepth24.log rdma_fio_bs512k_iodepth32.log rdma_fio_bs512k_iodepth64.log rdma_fio_bs512k_iodepth128.log rdma_fio_bs1024k_iodepth1.log rdma_fio_bs1024k_iodepth2.log rdma_fio_bs1024k_iodepth3.log rdma_fio_bs1024k_iodepth4.log rdma_fio_bs1024k_iodepth6.log rdma_fio_bs1024k_iodepth8.log rdma_fio_bs1024k_iodepth12.log rdma_fio_bs1024k_iodepth16.log rdma_fio_bs1024k_iodepth24.log rdma_fio_bs1024k_iodepth32.log rdma_fio_bs1024k_iodepth64.log rdma_fio_bs1024k_iodepth128.log tcp_fio_bs4k_iodepth1.log tcp_fio_bs4k_iodepth2.log tcp_fio_bs4k_iodepth3.log tcp_fio_bs4k_iodepth4.log tcp_fio_bs4k_iodepth6.log tcp_fio_bs4k_iodepth8.log tcp_fio_bs4k_iodepth12.log tcp_fio_bs4k_iodepth16.log tcp_fio_bs4k_iodepth24.log tcp_fio_bs4k_iodepth32.log tcp_fio_bs4k_iodepth64.log tcp_fio_bs4k_iodepth128.log tcp_fio_bs8k_iodepth1.log tcp_fio_bs8k_iodepth2.log tcp_fio_bs8k_iodepth3.log tcp_fio_bs8k_iodepth4.log tcp_fio_bs8k_iodepth6.log tcp_fio_bs8k_iodepth8.log tcp_fio_bs8k_iodepth12.log tcp_fio_bs8k_iodepth16.log tcp_fio_bs8k_iodepth24.log tcp_fio_bs8k_iodepth32.log tcp_fio_bs8k_iodepth64.log tcp_fio_bs8k_iodepth128.log tcp_fio_bs16k_iodepth1.log tcp_fio_bs16k_iodepth2.log tcp_fio_bs16k_iodepth3.log tcp_fio_bs16k_iodepth4.log tcp_fio_bs16k_iodepth6.log tcp_fio_bs16k_iodepth8.log tcp_fio_bs16k_iodepth12.log tcp_fio_bs16k_iodepth16.log tcp_fio_bs16k_iodepth24.log tcp_fio_bs16k_iodepth32.log tcp_fio_bs16k_iodepth64.log tcp_fio_bs16k_iodepth128.log tcp_fio_bs32k_iodepth1.log tcp_fio_bs32k_iodepth2.log tcp_fio_bs32k_iodepth3.log tcp_fio_bs32k_iodepth4.log tcp_fio_bs32k_iodepth6.log tcp_fio_bs32k_iodepth8.log tcp_fio_bs32k_iodepth12.log tcp_fio_bs32k_iodepth16.log tcp_fio_bs32k_iodepth24.log tcp_fio_bs32k_iodepth32.log tcp_fio_bs32k_iodepth64.log tcp_fio_bs32k_iodepth128.log tcp_fio_bs64k_iodepth1.log tcp_fio_bs64k_iodepth2.log tcp_fio_bs64k_iodepth3.log tcp_fio_bs64k_iodepth4.log tcp_fio_bs64k_iodepth6.log tcp_fio_bs64k_iodepth8.log tcp_fio_bs64k_iodepth12.log tcp_fio_bs64k_iodepth16.log tcp_fio_bs64k_iodepth24.log tcp_fio_bs64k_iodepth32.log tcp_fio_bs64k_iodepth64.log tcp_fio_bs64k_iodepth128.log tcp_fio_bs128k_iodepth1.log tcp_fio_bs128k_iodepth2.log tcp_fio_bs128k_iodepth3.log tcp_fio_bs128k_iodepth4.log tcp_fio_bs128k_iodepth6.log tcp_fio_bs128k_iodepth8.log tcp_fio_bs128k_iodepth12.log tcp_fio_bs128k_iodepth16.log tcp_fio_bs128k_iodepth24.log tcp_fio_bs128k_iodepth32.log tcp_fio_bs128k_iodepth64.log tcp_fio_bs128k_iodepth128.log tcp_fio_bs256k_iodepth1.log tcp_fio_bs256k_iodepth2.log tcp_fio_bs256k_iodepth3.log tcp_fio_bs256k_iodepth4.log tcp_fio_bs256k_iodepth6.log tcp_fio_bs256k_iodepth8.log tcp_fio_bs256k_iodepth12.log tcp_fio_bs256k_iodepth16.log tcp_fio_bs256k_iodepth24.log tcp_fio_bs256k_iodepth32.log tcp_fio_bs256k_iodepth64.log tcp_fio_bs256k_iodepth128.log tcp_fio_bs512k_iodepth1.log tcp_fio_bs512k_iodepth2.log tcp_fio_bs512k_iodepth3.log tcp_fio_bs512k_iodepth4.log tcp_fio_bs512k_iodepth6.log tcp_fio_bs512k_iodepth8.log tcp_fio_bs512k_iodepth12.log tcp_fio_bs512k_iodepth16.log tcp_fio_bs512k_iodepth24.log tcp_fio_bs512k_iodepth32.log tcp_fio_bs512k_iodepth64.log tcp_fio_bs512k_iodepth128.log tcp_fio_bs1024k_iodepth1.log tcp_fio_bs1024k_iodepth2.log tcp_fio_bs1024k_iodepth3.log tcp_fio_bs1024k_iodepth4.log tcp_fio_bs1024k_iodepth6.log tcp_fio_bs1024k_iodepth8.log tcp_fio_bs1024k_iodepth12.log tcp_fio_bs1024k_iodepth16.log tcp_fio_bs1024k_iodepth24.log tcp_fio_bs1024k_iodepth32.log tcp_fio_bs1024k_iodepth64.log tcp_fio_bs1024k_iodepth128.log ","date":"2024-07-19T22:36:00+08:00","permalink":"https://charlie0129.github.io/blog/p/nfs-over-rdma/","title":"NFS over RDMA Setup \u0026 Benchmark"},{"content":"10G ConnectX-4 Lx NICs can be flashed to 25G with the right firmware. This is useful if you have a 10G NIC and want to upgrade to 25G without buying a new NIC. This article will guide you through the process of flashing a 10G Mellanox ConnectX-4 Lx NIC to 25G.\nWarning: Do not try this on MCX4121C. It will brick the NIC.\nDownload firmware from Firmware for ConnectX®-4 Lx EN. Choose MCX4121A-ACUT. I will assume the firmware file is named fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACU_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin.\nIf you are wondering what\u0026rsquo;s the difference between models, here\u0026rsquo;s a table for ConnectX-4 Lx cards (original document). You should choose 25G models to flash to.\nMax Network Speed Interface Type Supported Ethernet Speeds (GbE) Host Interface Additional Features OPN 1x 10GbE SFP28 10, 1 PCIe 3.0 x8 MCX4111A-XCAT 2x 10GbE SFP28 10, 1 PCIe 3.0 x8 MCX4121A-XCAT 2x 10GbE SFP28 10, 1 PCIe 3.0 x8 Host Management, UEFI Enabled MCX4121A-XCHT 1x 25GbE SFP28 25, 10, 1 PCIe 3.0 x8 MCX4111A-ACAT 1x 25GbE SFP28 25, 10, 1 PCIe 3.0 x8 UEFI Enabled MCX4111A-ACUT 2x 25GbE SFP28 25, 10, 1 PCIe 3.0 x8 MCX4121A-ACAT 2x 25GbE SFP28 25, 10, 1 PCIe 3.0 x8 UEFI Enabled MCX4121A-ACUT 1x 40GbE QSFP28 40, 25, 10, 1 PCIe 3.0 x8 MCX4131A-BCAT 1x 50GbE QSFP28 50, 40, 25, 10, 1 PCIe 3.0 x8 MCX4131A-GCAT Download and install NVIDIA Firmware Tools (MFT).\nmst start\nmst ststus You will see you MST device, e.g. /dev/mst/mt4117_pciconf0. I will use this device path in this article. If your path differs from this, please change it accordingly.\nSave GUID, MAC: flint -d /dev/mst/mt4117_pciconf0 query full \u0026gt; flint_query.txt\nSave hardware info: flint -d /dev/mst/mt4117_pciconf0 hw query \u0026gt; flint_hwinfo.txt\nSave current firmware: flint -d /dev/mst/mt4117_pciconf0 ri orig_firmware.bin\nSave current firmware config: flint -d /dev/mst/mt4117_pciconf0 dc orig_firmware.ini\nSave current PXE ROM (if exists): flint -d /dev/mst/mt4117_pciconf0 rrom orig_rom.bin\nSave current PCI VPD: mlxburn -d /dev/mst/mt4117_pciconf0 -vpd \u0026gt; orig_vpd.txt\nflint -i fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACU_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin verify\nflint -i fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACU_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin -d /dev/mst/mt4117_pciconf0 -allow_psid_change burn\nReboot you machine. You should see your NIC is now have 25G speed.\nBonus:\nTo change MAC address: flint -d /dev/mst/mt4117_pciconf0 -mac 02c90abcdef0 sg To change GUID: flint -d /dev/mst/mt4117_pciconf0 -guid 0002c9000abcdef0 sg ","date":"2024-07-19T16:37:00+08:00","permalink":"https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/","title":"Flash 10G CX4121A to 25G"},{"content":"We have a 200Gbps InfiniBand network among about 100 nodes, connected using NVIDIA ConnectX-7 NICs (HDR200) to a NVIDIA MQM9700 switch. Each OSFP twin port (2xNDR, 2x400Gbps) on QM9700 is splitted into 4xNDR200 ports to connect to 4xNVIDIA ConnectX-7 NICs using 2xNDR to 4xNDR200 DAC/ACC (OSFP to 4x OSFP).\nWe want to test the performance of the network between 2 nodes just to make sure it is working. We expect to see a bandwidth of around 200Gbps.\nCheck the status of the Infiniband devices. Make sure at least one link is up (phys state: 5: LinkUp) on both nodes.\n1 2 3 4 5 6 7 8 9 root@tj01-h20-node139:/# ibstatus Infiniband device \u0026#39;mlx5_7\u0026#39; port 1 status: default gid:\tfe80:0000:0000:0000:\u0026lt;redacted\u0026gt; base lid:\t0x6c sm lid:\t0x4 state:\t4: ACTIVE phys state:\t5: LinkUp rate:\t200 Gb/sec (2X NDR) link_layer:\tInfiniBand Check if both nodes are connected to the Subnet Manager. You should see the name of both nodes appear on the list.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 root@tj01-h20-node139:/# ibnetdiscover # # Topology file: generated on Fri Jul 19 02:49:25 2024 # # Initiated from node \u0026lt;redacted\u0026gt; port \u0026lt;redacted\u0026gt; vendid=0x2c9 devid=0xd2f2 sysimgguid=0x9c05\u0026lt;redacted\u0026gt; switchguid=0x9c05\u0026lt;redacted\u0026gt;(9c05\u0026lt;redacted\u0026gt;) Switch\t129 \u0026#34;S-9c05\u0026lt;redacted\u0026gt;\u0026#34;\t# \u0026#34;MF0;CNTSN-POD229-QM97-IB-02:MQM9700/U1\u0026#34; enhanced port 0 lid 1 lmc 0 [1]\t\u0026#34;H-b83f\u0026lt;redacted\u0026gt;\u0026#34;[1](b83f\u0026lt;redacted\u0026gt;) # \u0026#34;tj01-4090-node005 mlx5_2\u0026#34; lid 22 2xNDR [2]\t\u0026#34;H-946d\u0026lt;redacted\u0026gt;\u0026#34;[1](946d\u0026lt;redacted\u0026gt;) # \u0026#34;tj01-4090-node004 mlx5_2\u0026#34; lid 27 2xNDR [3]\t\u0026#34;H-a088\u0026lt;redacted\u0026gt;\u0026#34;[1](a088\u0026lt;redacted\u0026gt;) # \u0026#34;tj01-4090-node006 mlx5_2\u0026#34; lid 10 2xNDR [4]\t\u0026#34;H-946d\u0026lt;redacted\u0026gt;\u0026#34;[1](946d\u0026lt;redacted\u0026gt;) # \u0026#34;tj01-4090-node007 mlx5_2\u0026#34; lid 12 2xNDR [6]\t\u0026#34;H-946d\u0026lt;redacted\u0026gt;\u0026#34;[1](946d\u0026lt;redacted\u0026gt;) # \u0026#34;tj01-4090-node009 mlx5_2\u0026#34; lid 11 2xNDR ...omitted Make sure perftest is installed.\nOn one node:\n1 root@tj01-4090-node099:/# ib_send_bw --report_gbit -a -F -d mlx5_2 --report_gbit: Show result in gigabit. -a: Run sizes from 2 till 2^23. -F: Do not show a warning even if cpufreq_ondemand module is loaded, and cpu-freq is not on max. -d: Use IB device. To make sure you are using the device you want to test (if you have multiple IB devices like me). On another node:\n1 root@tj01-h20-node139:/# ib_send_bw --report_gbit -a -F -d mlx5_2 tj01-4090-node099 tj01-4090-node099 is the IP address (hostname) of the other node.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 --------------------------------------------------------------------------------------- Send BW Test Dual-port : OFF\tDevice : mlx5_2 Number of qps : 1\tTransport type : IB Connection type : RC\tUsing SRQ : OFF PCIe relax order: ON ibv_wr* API : ON TX depth : 128 CQ Moderation : 100 Mtu : 4096[B] Link type : IB Max inline data : 0[B] rdma_cm QPs\t: OFF Data ex. method : Ethernet --------------------------------------------------------------------------------------- local address: LID 0x6c QPN 0x0050 PSN 0xddd611 remote address: LID 0x28 QPN 0x0050 PSN 0xa3c746 --------------------------------------------------------------------------------------- #bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps] 2 1000 0.052812 0.050794 3.174618 4 1000 0.080899 0.080547 2.517093 8 1000 0.17 0.17 2.624748 16 1000 0.35 0.35 2.741373 32 1000 0.70 0.70 2.736427 64 1000 1.62 1.62 3.159528 128 1000 3.07 3.07 2.994503 256 1000 5.46 5.46 2.663988 512 1000 13.24 13.23 3.230121 1024 1000 26.58 26.56 3.241834 2048 1000 51.80 51.75 3.158744 4096 1000 82.15 82.11 2.505647 8192 1000 89.37 89.33 1.363014 16384 1000 89.01 87.24 0.665597 32768 1000 172.67 96.54 0.368268 65536 1000 181.95 108.83 0.207586 131072 1000 181.82 137.78 0.131399 262144 1000 187.32 163.15 0.077797 524288 1000 190.12 179.08 0.042696 1048576 1000 191.18 187.28 0.022326 2097152 1000 192.69 192.69 0.011485 4194304 1000 195.53 195.53 0.005827 8388608 1000 196.69 196.69 0.002931 --------------------------------------------------------------------------------------- ","date":"2024-07-19T11:10:00+08:00","permalink":"https://charlie0129.github.io/blog/p/ib-perf-test/","title":"InfiniBand Performance Test"},{"content":"Note Note that this blog currently acts as a reference for me. It is not a comprehensive guide on why and how to use Nix on macOS.\nI chose nix because I am tired of Homebrew\u0026rsquo;s slowness and NOT PROVIDING BOTTLES (BINARY) FOR OLD MACOS VERSIONS.\nInstall I am not using the official installer because it cannot survive macOS updates. Instead, I am using the Determinate System\u0026rsquo;s Nix Installer.\n1 2 3 4 5 6 7 ARCH=$(uname -m | sed \u0026#39;s/arm64/aarch64/\u0026#39;) OS=$(uname -s | tr \u0026#39;[:upper:]\u0026#39; \u0026#39;[:lower:]\u0026#39;) curl -L -o nix-installer https://github.com/DeterminateSystems/nix-installer/releases/latest/download/nix-installer-$ARCH-$OS chmod +x nix-installer # Optionally, move nix-installer to your PATH # sudo install nix-installer /usr/local/bin ./nix-installer install --explain Pin Nixpkgs I don\u0026rsquo;t want to download a tarball and extract it every now and then. I know you can set tarball-ttl to a higher value, but I don\u0026rsquo;t want to do that either. Just pin nixpkgs.\n1 nix registry pin nixpkgs If you are like me in China, you may want to use a mirror.\n1 echo \u0026#34;substituters = https://mirrors.sjtug.sjtu.edu.cn/nix-channels/store/ https://cache.nixos.org/\u0026#34; | sudo tee -a /etc/nix/nix.conf Using I wrote some scripts just to make my life easier.\nnix-install nix-list nix-search nix-uninstall Later Steps There is nix-darwin to make declarative configuration easier. I don\u0026rsquo;t have time to try it yet but I will definitely give it a try when I have time.\n","date":"2024-06-27T15:13:00+08:00","permalink":"https://charlie0129.github.io/blog/p/macos-nix-first-taste/","title":"macOS Nix First Taste"},{"content":"前言 由于最近接入了 10Gbps 网络，发现在 PT 做种时，传输速度并没有达到预期的速度，于是优化了一波。本文记录了一些针对高速做种的建议、注意点和技巧。\n硬件条件 足够快的网络 做种本质上就是网络传输，快速的网络是做种的基础。\n确保上传带宽 做种嘛，主要是给别人传输数据，吃的是你的上传带宽，跟下载几乎没有关系，这是最重要的。例如原本 50Mbps 的上传情况下，买一条 100Mbps 的宽带比以下所有的优化都重要。\n但是在中国，一般来说家庭宽带的上行速度都是比较低的。例如，电信的 1000M 宽带，上行速度一般不会超过 100Mbps，通常在 50Mbps 左右。这样的话，你的上传速度就被限制在 6.25MB/s 左右，在这个速度下，其实最主要的瓶颈在于你的上传带宽上，不太需要下面说的硬件/软件优化🫠。你可能需要找不少资源来找到高上传带宽的宽带，例如某些地区的移动宽带会给 IPv6 相当大的上传。你还要注意如何不被运营商认为是 PCDN 而封你的号。\n由于本文聊的是高速做种，针对的是接近 1Gbps 的上传带宽的用户。国外的情况暂且不讨论（高速宽带价格更低，很容易购买到超过 1Gbps 的云服务器、seeding box 等），在国内，拥有高速上传带宽的用户主要有几种情况：\n企业用户（例如大型公司、数据中心等），使用价格高昂，上下对等（即下载和上传速度一样快），租云服务器也属于这个范畴； 高校用户，使用价格较低（对用户来说，对学校来说不是这回事），上下对等； 其他特殊用户，例如某些地区的移动家庭宽带，500M 宽带上行可以达到 200M。 如果你是这些用户，那么你的上传带宽可能会达到 1Gbps 甚至更高，这时候就需要考虑下面的硬件/软件优化了。不过估计阅读本文的人大部分是高校教育网用户，低廉的价格超大的上传带宽（\n在本文中，我的接入网络为上下 10Gbps 的教育网 CERNET 对等接入。\n外界可联通 大部分情况下，你做种时是别人主动连接你的，所以你的网络需要是可被连接的。\n对于 IPv4 来说：\n最好的情况是你拥有一个独立的公网 IP ，这是最简单的情况。如果公网 IP 在你的网卡上，那么你不需要做任何设置。如果公网 IP 在你家的光猫上，那么通过简单的端口映射/UPnP 等就能完成。这种情况国内不常见。 其次如果你的网络是 NAT1（全锥形 NAT） 的（指国内运营商级 NAT，一般你的光猫上能获取 100.x.x.x 这样的 IP），那么通过一些复杂的奇技淫巧（例如 UDP hole punching）也能实现可被连接。 最差的情况是你的网络是 NAT4（对称形 NAT） 的，这种情况下，你的网络是不可被连接的，除非你使用一些中继服务器来帮助你穿透。不过这种情况下，因为中继服务器的带宽有限，通常不能达到高速做种的要求，这里不讨论。 如何判断你是 NAT1 还是 NAT4 ？你可以使用 pystun3。\n如果你使用 IPv6 做种（常见于教育网），运营商（学校）下发的 IPv6 地址一般都是公网 IP，不需要做任何设置，等同于情况一。\n如果你不确定自己的网络是否可被连接，可以阅读文末注意点。\nPeering 在数 Gbps 的高速网络下，你所在的运营商网络和你的种子的 peer 之间的连通性也值得考虑（但没那么重要）。运营商各张网之间，例如电信的 163 骨干网，教育网 CERNET ，甚至国外的运营商等，他们之间互联需要走 BGP ，如果你和你需要连接的用户之间连通性不好，那么你的上传速度可能会受到影响。\n国内一般不需要考虑，这种一般在跨国传输的情况下比较明显，如果你所在的网的国际出口带宽不够，连接国外的 peer 时你会发现你的上传速度上不去。\n足够快的硬盘 重要性仅次于网络的是硬盘速度，尤其关注 随机读。一种常见的错误认知是：我的机械硬盘顺序读能有 200MiB/s 那么我就能吃满 1Gbps 的上传带宽应该轻轻松松吧。这是完全错误的，由于做种会产生大量的 随机读 ，机械硬盘在这方面是最弱鸡的。\n例如，在 1Gbps 左右上传，大文件做种（活跃数据超过我的内存大小 128GB ）的情况下，能稳定 10k IOPS 直达我的磁盘的读取（例如下图 14k IOPS 的读）。一块 7200 RPM 的 SATA 机械硬盘也就 100 IOPS ，一块 15000RPM 的 SAS 机械硬盘也就 200 IOPS 左右，这有着快 2 个数量级的差距。所以，如果你的硬盘是机械硬盘（HDD, spinning rust :p），建议放弃治疗，一旦内存缓存不够了你的上传速度就 gg 了，这辈子都不可能吃满 1Gbps 的，500Mbps 都够呛。根据我的经验， 3*8TB 的 7200RPM 企业级 HDD RAID5 在活跃数据超过内存缓存（128GB）的情况下也就勉勉强强能吃 200Mbps 的上传带宽（此时磁盘已经 100% busy 了），如果缓存更小的话这个数值甚至更低。\n注意：这里不考虑完全使用内存缓存（因为大部分人都没有几百 GB 的内存）一旦种子活跃数据超过可用的内存缓存大小，硬盘就是你的上传速度的瓶颈。\n所以高速做种的场景下一定要用 SSD ，最次也需要是 SATA SSD，建议使用更高性能的 NVME SSD 。\n本文使用了 3 块 Micron 5400 PRO 1.92TB 的 SSD 组成的 RAID5 。注意，这对于我的网络接入（10 Gbps）的情况下仍然是远远不够的，正常需要高性能 NVME SSD 才行。不过我大部分情况也就 1Gbps 左右，跑不满 10Gbps ，凑合用吧。\n如果你不确定你的硬盘是否足够快，那么你可以阅读下文判断磁盘 IO 瓶颈的部分。\n足够大的内存 足够大的内存可以提供：\n更大的磁盘缓存，减少磁盘压力（高速上传时，SSD 也不一定能跟上）； 更大的 send buffer ，提高网络传输效率/IO 吞吐； 一般来说 32GB 左右能够满足需求，本文中使用了 128GB 内存测试。\n足够快的网卡 更好的网卡能够提供更多的 hardware offloading ，减轻 CPU 的负担，提高网络传输效率。一些高端网卡也支持更多的 queue ，提高并发传输能力，例如 CX-4 Lx 支持 64 个 queue ，可以看到这些 queue 上的收发包表示它们在被积极利用（实际上一般软件无法同时用不到这么多，包括 qBittorrent，所以 8 队列的 I350 就够了）。\n最差也要使用 Intel 的 Gigabit 网卡，例如 I350 、I210 ，更好的可以用 Intel X520/X540 ，甚至 Mellanox ConnectX-4/5/6 等更高端的网卡。不要用 Realtek （小螃蟹）的，在大连接数吃满 1Gbps 的情况下性能很烂。\n例如下图是我的计算机在 700Mbps 下行 1.2Gbps 上行时每秒的收发包数量，约为 170 Kpps 。性能差的网卡是很难达到这个性能的。\n本文中使用了 Mellanox ConnectX-4 Lx 25G (CX4121A) 网卡（虽然现在被 NVIDIA 收购了，不过我还是习惯称之为 Mellanox ）。\n足够快的 CPU 一个正常的现代 CPU 即可，例如 Intel Core i5-12500 。请不要使用树莓派、老掉牙的 CPU（例如 Intel Xeon E5-2403 v2 这种 4C4T 1.8GHz 的垃圾），这种情况下，你的 CPU 仍然是整个系统的瓶颈，特别是在高速上传时，CPU 的负载会很高。网卡越烂，上传速度越快，你就需要越好的 CPU 来处理。\n本文中使用了国产的飞腾 2000+ ，虽然看起来有 64 个 ARM64 核心，但是性能还是比较烂的，不太够说实话，用起来跟只有 10 个核的 i9-10900K 也差的比较多。\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 Architecture: aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian Address sizes: 44 bits physical, 48 bits virtual CPU(s): 64 On-line CPU(s) list: 0-63 Thread(s) per core: 1 Core(s) per socket: 64 Socket(s): 1 NUMA node(s): 8 Vendor ID: Phytium BIOS Vendor ID: Phytium Model: 2 Model name: FTC662 BIOS Model name: FT-2000+/64 Stepping: 0x1 BogoMIPS: 100.00 L1d cache: 2 MiB L1i cache: 2 MiB L2 cache: 512 MiB NUMA node0 CPU(s): 0-7 NUMA node1 CPU(s): 8-15 NUMA node2 CPU(s): 16-23 NUMA node3 CPU(s): 24-31 NUMA node4 CPU(s): 32-39 NUMA node5 CPU(s): 40-47 NUMA node6 CPU(s): 48-55 NUMA node7 CPU(s): 56-63 Flags: fp asimd evtstrm crc32 cpuid 软件调优 qBittorrent 基本设置 提升连接限制。如果你的网络/磁盘足够强劲，你可以去除所有的连接限制： 提高队列。磁盘性能足够的情况下，可以同时让多个种子做种，你也可以直接关闭： 提高内存限制。虽然目前（2024 年 6 月）这个设置只对 Windows 有效，不过设置了也无妨： libtorrent 调优 我们针对 libtorrent \u0026gt;=2.0 版本，1 Gbps 左右的对等带宽， 64G 左右内存的用户，其他用户请酌情调整以下参数。\naio_threads Asynchronous I/O threads 默认 10 。例如在高速下载时，由于需要把下载的数据都坐 SHA1/SHA256 校验，CPU 可能存在瓶颈。提高这个值可以减轻让多核 CPU 并行处理。常见的说法是设置为 4 *线程数 。由于我使用了 64 核的处理器，256 显得过大了，所以我设置为 64 。 file_pool_size File pool size 默认 40。一个 session 直接同时打开多少个文件。大量做种时，这个值设置的太小会导致频繁的文件打开/关闭，降低性能。而又设置的太大会超过一个进程的文件描述符限制（ulimit -n），导致打开文件失败。我设置为 900 （Linux 默认的文件描述符限制是 1024），留一些给其他的文件使用。 max_queued_disk_bytes Disk queue size 默认 1024 KiB 。这个值表示磁盘 IO 写入队列的大小，如果达到了这个上限，libtorrent 就会停止下载，直到磁盘完成了一部分写入。这个值设置的太小会导致下载性能下降，设置的太大会导致内存占用过高。我设置为 262144 KiB 。 default-disk-io-constructor Disk IO type 默认 mmap_disk_io_constructor 。这个值表示 libtorrent 使用的磁盘 IO 类型，posix_disk_io_constructor 是使用 POSIX IO ，mmap 是使用 mmap ，mmap 在 kernel 里有大量优化，建议只用 mmap （Memory Mapped Files）。如果你看到了大量内存占用，请看文末注意点。 disk_io_write_mode 和 disk_io_read_mode 默认 enable_os_cache 。维持默认。 piece_extent_affinity Use piece extent affinity 默认 0 。这个值表示 libtorrent 会尽量下载相邻的 piece ，提高磁盘 IO 吞吐。建议打开。 suggest_mode Send upload piece suggestions 默认 no_piece_suggestions 。这个值表示会告诉从我这边下载的 peer 建议下载那些在我读缓存里的 piece 而不是其他的没在缓存里的，可以提高做种速度减少磁盘 IO 。建议打开。 send_buffer_low_watermark Send buffer low watermark 默认 10 KiB。最小的目标 send buffer 大小（包括等待磁盘读取的字节数）。这实际上是初始窗口大小，它决定了我们能够多快地提高发送速率。我设置为 4096 KiB。 send_buffer_watermark Send buffer watermark 默认 500 KiB。如果发送缓冲区的字节数少于它，将从磁盘额外读取 16 KiB 的数据到缓冲区。这是上限，所以实际大小可能小于它。过小影响上传速率，过大浪费内存。我设置为 32768 KiB。 send_buffer_watermark_factor Send buffer watermark factor 默认 50。这是个百分比，peer 的上传速率乘以这个因子以获得 实际的 send_buffer_watermark，但是不超过上面的 send_buffer_watermark。对于高速连接，设置得更高可以提高上传性能和磁盘吞吐，设置得过高可能会浪费内存并且偏好读。我的目的是高速做种，我设置为 200。 connection_speed Outgoing connections per second 默认 30。这个值表示每秒最多建立多少个出站连接，影响去下载 peer 那边数据的效率。我设置为 500 。 listen_queue_size Socket backlog size 默认 5 。它是传递给监听 socket listen() 的值，用于指定未处理传入连接的队列长度。当我们不积极等待连接被接受时，这些连接将会被排队。由于我们是大带宽高性能做种，我设置为 3000。 mixed_mode_algorithm μTP-TCP mixed mode algorithm 默认 prefer_tcp 。这个值用于确定在同时存在 TCP 和 uTP 连接时如何处理这两种连接。我们不需要为 uTP 做出让步，所以设置为 prefer_tcp 。 allow_multiple_connections_per_ip Allow multiple connections from the same IP address 默认 true 。这个值表示是否允许同一个 IP 建立多个连接。我们不需要限制，所以设置为 true 。 choking_algorithm Upload slots behavior 使用 fixed_slots_choker。 seed_choking_algorithm Upload choking algorithm 使用 fastest_upload。 max_out_request_queue Maximum outstanding requests to a single peer 默认 500 。表示发给 peer 的最大未处理请求数。我设置为 2000 。 自动下载/自动 Announce 一般种子流量最大的时候为刚发布之后，为了在种子发布后能够快速下载，抢占最早的上传机会。我们需要自动下载种子，然后立即开始做种。称之为 racing 。\n这里我使用了自己编写的脚本，你也可以用 autodl-irssi 等工具。\n注意点 判断磁盘 IO 瓶颈 使用 iostat -xm 1，找到存放种子数据的盘，特别关注最后的百分比，接近 100% 表示你的磁盘是瓶颈了。\n你也可以在 htop 中看到 IO 的情况（在 Setup 的 Meter 中添加 Disk IO ）。htop 中的百分比是所有磁盘的合计，如果的下载盘是软 RAID 你有可能看到超过 100% 的情况，我这里是单个设备（硬 RAID），所以图中 99.1% 表示 IO 已经满了，我的硬盘已经跟不上了。\n上面两张图是我使用的 3 块 SSD 阵列在 1~2 Gbps 上传，IO 爆发时的情况，可以看到其实 IO 已经快满了。如果你发现你的 busy% 接近 100% 了，那么你的磁盘 IO 就是瓶颈了，建议更换更好更快的 SSD。\n如果你用的是机械硬盘，应该是轻轻松松全程 100% ，完全撑不住 :p\n判断网络 IO 瓶颈 首先查看你的上传/下载速度是否超过了你的互联网带宽/物理网络接口的速度。你可以在 htop 中看到（记得在 Setup -\u0026gt; Meters 中添加 Network IO ）如果超过了，那么没什么办法，除非换网络/换网卡。如果没有，可以继续往下看。\n如果你看到 softirq 占用过高，那么你的网络 IO 可能是瓶颈了，建议更换更好的 CPU 或网卡。\n例如你可以在 htop 中看到紫色的 softirq 的占用（注意在 Setup -\u0026gt; Display Options 中勾选 Detailed CPU time ）。\n如果你发现某个/几个核心的 softirq 占满了，那么你的网络 IO 可能是瓶颈了。\n正常情况你不应该看到很高的 softirq 占用。\n判断 CPU 瓶颈 一般来说，通过看 htop 即可。注意，你不仅要看全部的 CPU 核心占用，还要看单个核心的占用。如果你发现某个核占满了（这种情况更易发生）、整体 CPU 占用很高（目前 CPU 核心很多了，这个不太容易发生），那么你的 CPU 可能是瓶颈了。同时，看到 CPU 占用很高时，注意上面一节提到的 softirq 占用，这可能是网络 IO 的瓶颈。\n例如在我的计算机上，我偶尔在 IO 爆发时能见到 CPU 占用 5000%+ 的峰值，这说明我的CPU对我的使用场景来说不太行。（图中其实你会发现我的磁盘IO也满了。。）\nqBittorrent 内存占用过大？ 如果你发现 qBittorrent 的内存占用过大，你不应该担心。这是 mmap 的特性，它会把文件映射到内存中，这样可以减少磁盘 IO ，提高性能。如果你关注总的内存占用，你会发现它并不会增加。自从 libtorrnet 2.0 使用 mmap 以来，有很多用户 反馈内存占用过大 ，但是这是正常的。\n例如我的计算机上，qBittorrent 的内存占用超过了 80GB ，但是总的内存占用只有 5GB。\n检测是否可被连接 你可以使用外网 TCPing 来检查，例如一些网站可以替你检测： IPv4、IPv6 。输入你的公网 IP 和端口，如果各地均能够 ping 通，那么你是可被连接的。\n上传仍然不及预期？ 如果你各方面都确认没有瓶颈（网络、磁盘、CPU 等）但是上传速度仍然很慢，那么可能是你的种子不够热门，或者说你的 peer 的下行太慢了。这方面除了选择热门的种子（做种人少，下载人多，提高下载的 peer 连接你的机会），其实也没什么办法了。\n因为你的上传是需要 peer 来连接你的，这就不是你能决定的事，可遇而不可求。举个例子，如果想跑满 10Gbps 的上传，那么需要 10 个 peer ，每个都以 1Gbps 的速度从你这拉数据，这是非常困难的。首先，别人的下载速度很少有 1G 的；其次，做种的人几十个，别人不可能只从你这下载，很可能从几十个不同的 peer 那边下载，一下就分散了，一个 peer 到你这只有几十分之一的流量了。\n所以，这种情况下，除了选择合适的种子提高 peer 连接你的机会也没什么办法了。合适的种子通常是刚发布、热门、做种人少、下载人多的种子。\n","date":"2024-06-19T22:11:00+08:00","permalink":"https://charlie0129.github.io/blog/p/high-speed-seeding-guide/","title":"高速做种指南"},{"content":" Recent Update: One month later, after we opened our issue #10264, it turns out that the gVisor developers are also aware of tha lacking of documentation about the systemd-cgroup option. Finally:\nThey have added the systemd-cgroup option to the documentation in 8c3abba, although the documentation website doesn\u0026rsquo;t seem to be updated as of now. runsc will throw a warning if it detects a systemd-like path but systemd-cgroup is not used to tell you about the possible misconfiguration. So if you are using gVisor with systemd cgroup, you should add the systemd-cgroup option to the runsc configuration, if not already.\nThe Problem Recently we found some of our Kubernetes nodes were constant going down, completely dead, with no luck connecting to it. Taints like node.kubernetes.io/unreachable:NoSchedule and node.kubernetes.io/unreachable:NoExecute were automatically added to the nodes because the kubelet was not able to communicate with the API server. The only way to bring the node back was to restart it. After some debugging, we found out that some bad gVisor-created (runsc runtime) Pods was eating all the memory of the node and killing it. Traditional Pods with runc runtime were running fine.\nYou may say, \u0026ldquo;But there are memory limit on a Pod. If it uses more than the limit it should be killed by the OOM killer, right\u0026rdquo;？ That\u0026rsquo;s what I thought too, so I applied the following Deployment to intentionally consume some memory and see if it will be killed.\nClick to see full yaml 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 apiVersion: apps/v1 kind: Deployment metadata: labels: app: memory-eater-bash name: memory-eater-bash namespace: default spec: replicas: 1 selector: matchLabels: app: memory-eater-bash template: metadata: labels: app: memory-eater-bash spec: containers: - command: - bash args: - -c - big_var=data; while true; do big_var=\u0026#34;$big_var$big_var\u0026#34;; done; sleep 2d image: python:3.12-bookworm name: ubuntu securityContext: readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 999 resources: limits: cpu: 100m memory: 256Mi requests: cpu: 100m memory: 200Mi dnsPolicy: Default hostNetwork: true restartPolicy: Always runtimeClassName: gvisor I applied 256MiB memory limit on it, hoping it will be kill after it uses more than 256MiB memory. But it didn\u0026rsquo;t. It kept eating all the memory until the node was dead, bypass OOM protection provided by Kubernetes. It ate over 500GiB of memory, which is the total memory of the node, and we were no longer able to connect to the node. Luckily, we eviction process kicked in and existing Pods on the node are moved to other nodes, so the service was not affected by much.\nThe Debug Process Of course, we need to apply the above memory-eater Deployment to reproduce the issue. But killing a node every time is not a good idea 😂 , especially when we need to manually restart it. So I used another Deployment to consume some (1GiB) memory, but not all of it.\nClick to see full yaml 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 apiVersion: apps/v1 kind: Deployment metadata: labels: app: memory-eater-python name: memory-eater-python namespace: default spec: replicas: 1 selector: matchLabels: app: memory-eater-python template: metadata: labels: app: memory-eater-python spec: containers: - command: - python args: - -c - big_list = []; print(\u0026#39;Allocating 1GB of memory...\u0026#39;); [big_list.append(\u0026#39; \u0026#39; * 10**6) for _ in range(1000)]; import time; time.sleep(100000) image: python:3.12-bookworm name: ubuntu securityContext: readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 999 resources: limits: cpu: 100m memory: 256Mi requests: cpu: 100m memory: 200Mi dnsPolicy: Default hostNetwork: true restartPolicy: Always runtimeClassName: gvisor Just as I expected, the memory limit did not work. Time to debug.\nTrying to reproduce Before I trying to do anything on the production cluster, I tried to reproduce the issue on my local machine. Just to be safe not to kill the production cluster and get fired :P (just kidding. It\u0026rsquo;s because debugging is a lot easier on my local machine).\nI used Docker with gVisor as per the official documentation. It was working fine. Our production cluster is using containerd, so was it a containerd issue? I was in a hurry so I used minikube with gVisor addon enabled, which should provide a Kubernetes cluster with containerd and gVisor (and a containerd shim for runsc). At the time of writing, it was Kubernetes v1.28.3, containerd v1.6.24, and gVisor release-20240401.0.\nI know, I know. We should keep versions of the software same as the production cluster. But I was in a hurry and I was not sure if the issue was with containerd or gVisor. So I just went with it. Spoiler: yes, it has something to do with versions. If I were to do it again, I would use the same versions as the production cluster.\n1 minikube start --addons=\u0026#34;gvisor\u0026#34; --container-runtime=containerd --driver=docker --cpus=6 I applied the memory-eater Deployment and waited some surprising results. But the 256MiB memory limit was working as expected. The Pod was killed after it used more than 256MiB memory. I smelled some fishy things going on in the production cluster.\nDebugging on the production cluster Again, I applied the memory-eater Deployment, but on the production cluster this time. Hoping it will not kill the node. To my relief, it did not kill the node. But it did not kill the Pod with its memory (969MiB) above limits (256MiB) either (which is expected).\n1 2 3 $ kubectl top pod memory-eater-python-67cc4dd4f7-sp2r6 NAME CPU(cores) MEMORY(bytes) memory-eater-python-67cc4dd4f7-sp2r6 0m 969Mi Before I dig anything deeper (like with some gVisor debugging tools), I checked the sandbox created by runsc (gVisor) to see if there is anything suspicious.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 # On production node $ PID=$(ps aux | grep memory-eater-python | grep runsc-sandbox | awk \u0026#39;{print $2}\u0026#39;) $ cat /proc/$PID/cmdline | tr \u0026#39;\\0\u0026#39; \u0026#39;\\n\u0026#39; runsc-sandbox --log-format=json --panic-log=/var/log/pods/default_memory-eater-python-74975bf59f-7fflh_1cf55082-d769-4a51-a424-2312c23eae82/gvisor_panic.log --root=/run/containerd/runsc/k8s.io --log=/run/containerd/io.containerd.runtime.v2.task/k8s.io/6a541285fa39d743dc20bd8b3169c4a8f5cd4d77542144fe82d3f6f7af469c2c/log.json --log-fd=3 --panic-log-fd=4 boot --apply-caps=false --bundle=/run/containerd/io.containerd.runtime.v2.task/k8s.io/6a541285fa39d743dc20bd8b3169c4a8f5cd4d77542144fe82d3f6f7af469c2c --controller-fd=10 --cpu-num=96 --io-fds=5,6 --mounts-fd=7 --overlay-mediums=0,0 --setup-root=false --spec-fd=11 --start-sync-fd=8 --stdio-fds=12,13,14 --total-host-memory=540886331392 --total-memory=540886331392 # 503GiB --user-log-fd=9 --product-name=AS -4124GS-TNR --proc-mount-sync-fd=22 6a541285fa39d743dc20bd8b3169c4a8f5cd4d77542144fe82d3f6f7af469c2c Hmm, something caught my eye.\n1 2 --total-host-memory=540886331392 --total-memory=540886331392 # 503GiB Why is the total-memory and total-host-memory set to 503GiB? That\u0026rsquo;s the total memory of the node. I assume it should be set to the memory limit of the Pod, which is 256MiB. To verify my assumption, I checked the sandbox created by runsc on my local machine and it indeed was set to 256MiB.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 # On my local machine $ PID=$(ps aux | grep memory-eater-python | grep runsc-sandbox | awk \u0026#39;{print $2}\u0026#39;) $ cat /proc/$PID/cmdline | tr \u0026#39;\\0\u0026#39; \u0026#39;\\n\u0026#39; runsc-sandbox --root=/run/containerd/runsc/k8s.io --log=/run/containerd/io.containerd.runtime.v2.task/k8s.io/f11a26742777348875e6c7f182d9f8cc5c4a1c3a7f726ad5e8cb956b0c2dea96/log.json --log-format=json --panic-log=/var/log/pods/default_memory-eater-python-fc75975c8-vncwz_480a4dd9-22ad-4e43-9ecb-b797735e990a/gvisor_panic.log --log-fd=3 --panic-log-fd=4 boot --apply-caps=false --bundle=/run/containerd/io.containerd.runtime.v2.task/k8s.io/f11a26742777348875e6c7f182d9f8cc5c4a1c3a7f726ad5e8cb956b0c2dea96 --controller-fd=10 --cpu-num=6 --dev-io-fd=-1 --gofer-mount-confs=lisafs:none,lisafs:none --io-fds=5,6 --mounts-fd=7 --setup-root=false --spec-fd=11 --start-sync-fd=8 --stdio-fds=12,13,14 --total-host-memory=2061119488 --total-memory=268435456 # 256MiB --user-log-fd=9 --proc-mount-sync-fd=22 f11a26742777348875e6c7f182d9f8cc5c4a1c3a7f726ad5e8cb956b0c2dea96 It has the correct limits on my local machine.\n1 2 --total-host-memory=2061119488 --total-memory=268435456 # 256MiB So the issue was with this --total-memory argument. But why is it incorrect on the production cluster?\nChecking the versions Seeing the difference, I checked the versions on my local cluster and the production cluster.\nTool Local Production Kubernetes 1.28.3 1.26.2 containerd v1.6.24 v1.7.0-rc.1 runsc 20240401.0 20231009.0 Both are running Ubuntu Server 22.04 LTS, using systemd+cgroupsv2.\nI assume the problem was with runsc (gVisor).\nWhat\u0026rsquo;s wrong with gVisor? Cgroups v2 support After I know the issue is with the --total-memory flag. I found there is a similar issue reported google/gvisor #9580. The author had great explanation of the issue:\nUnder Kubernetes + cgroups v2 + systemd, gVisor launches all processes into the container subgroup associated with the pause container. This makes some sense given that cgroups v2 specifies that processes can only exist at leaf nodes, and the pod\u0026rsquo;s cgroup is registered as a slice (an intermediate unit which cannot have its own processes) with systemd. When the sandbox is launched gVisor needs a container subgroup and the pause container is the first to be launched. The pause container is a child of the pod cgroup and therefore inherits the limits of the parent pod cgroup, BUT the child\u0026rsquo;s controllers reflect the default max value. This in turn means that this code which reads the memory limit and cpu quota reads these as unlimited.\nLet me explain a bit more. So a normal cgroup v2 hierarchy for a Pod looks like this:\n1 2 3 4 /sys/fs/cgroup/kubepods.slice/ ├── kubepods-burstable.slice │ └── kubepods-burstable-pod\u0026lt;pod_id\u0026gt;.slice # has memory limit │ └── cri-containerd-\u0026lt;container_id\u0026gt;.scope # no memory limit The pod slice has a memory limit (cat memory.max gives xxx), but the container scope does not (cat memory.max gives max). The container scope is where the container is running. The container scope is a child of the pod slice. So the container should inherit the memory limit of the pod slice. This is how cgroups works.\nBut gVisor only checks the container scope and says \u0026ldquo;oh, it has no memory limit. Let\u0026rsquo;s set the total memory to max (the total memory of the node)\u0026rdquo;. And that\u0026rsquo;s why the --total-memory is set to 503GiB on the production cluster.\nThis issue was fixed in google/gvisor #9631. The code below is from the PR which fixed the issue (after the comments). So it checks the Pos slice (parent) if the memory limit is not set in the container slice (leaf node).\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 // File: runsc/cgroup/cgroup_v2.go // Link: https://github.com/google/gvisor/blob/43c2c00c5006e7d21ce9cd431692622031891231/runsc/cgroup/cgroup_v2.go#L333-L346 // MemoryLimit returns the memory limit. func (c *cgroupV2) MemoryLimit() (uint64, error) { limStr, err := getMemoryLimit(c.MakePath(\u0026#34;\u0026#34;)) if err != nil { return 0, err } // In cgroupv2+systemd, limits are set in the parent slice rather // than the leaf node. Check the parent to see if this is the case. if limStr == maxLimitStr { parentLimStr, err := getMemoryLimit(filepath.Dir(c.MakePath(\u0026#34;\u0026#34;))) if err != nil \u0026amp;\u0026amp; !errors.Is(err, os.ErrNotExist) { return 0, err } if parentLimStr != \u0026#34;\u0026#34; { limStr = parentLimStr } if limStr == maxLimitStr { return math.MaxUint64, nil } } return strconv.ParseUint(limStr, 10, 64) } The runsc binary on our production cluster was built before the fix was merged. So I assume the issue was fixed in the runsc binary after the merge.\nSystemd cgroup configuration So if the issue was fixed in the latest release, simply upgrading the runsc binary should fix the issue. No. The issue was still there. I checked the runsc binary on the production cluster and it was built after the fix was merged. So what\u0026rsquo;s wrong?\nRemember what a normal cgroup v2 hierarchy looks like in the previous section? Although the cgroups created by runc are just like that, that\u0026rsquo;s not how the ones created by runsc looks like on our production cluster. Instead, they look like this:\n1 2 3 4 5 6 /sys/fs/cgroup/ ├── kubepods.slice │ ├── kubepods-burstable.slice │ │ └── kubepods-burstable-pod\u0026lt;pod_id\u0026gt;.slice ├── system.slice │ ├── kubepods-burstable-pod\u0026lt;pod_id\u0026gt;.slice:cri-containerd:\u0026lt;container_id\u0026gt; Why are the container cgroups not under the pod cgroups? Why are they in system.slice? If that was the case, the fix above won\u0026rsquo;t do anything because the parent slice is not the pod slice. The parent slice is system.slice, it has no memory limit. So the container will have no memory limit even after the fix. OOM\u0026rsquo;in the node.\nAfter some issue-searching on the gVisor GitHub repo, I found that runsc initially has no support for systemd cgroups google/gvisor #193. Support for systemd cgroup was added in google/gvisor #7287. It required a specific configuration in the runsc configuration:\n1 2 3 4 // File: runsc/config/config.go // Link: https://github.com/google/gvisor/blob/bf86207401cab99d859b25acd4911038608f0d33/runsc/config/config.go#L229-L230 +\t// Use systemd to configure cgroups. +\tSystemdCgroup bool `flag:\u0026#34;systemd-cgroup\u0026#34;` Hmm, looks like there was nothing like that in the runsc configuration on the production cluster. Go! Fix it! Find one of the cordoned node, drain it, upgrade the runsc binary, and change the configuration.\n1 2 3 4 5 6 7 8 9 10 /etc/containerd/config.toml [plugins.\u0026#34;io.containerd.grpc.v1.cri\u0026#34;.containerd.runtimes.runsc] runtime_type = \u0026#34;io.containerd.runsc.v1\u0026#34; + [plugins.\u0026#34;io.containerd.grpc.v1.cri\u0026#34;.containerd.runtimes.runsc.options] + TypeUrl = \u0026#34;io.containerd.runsc.v1.options\u0026#34; + ConfigPath = \u0026#34;/etc/containerd/runsc.toml\u0026#34; /etc/containerd/runsc.toml\u0026#34; + [runsc_config] + systemd-cgroup = \u0026#34;true\u0026#34; Restart containerd systemctl restart containerd. Apply a new Pod with toleration and nodeSelector to schedule it to a cordoned node. And it worked! The container cgroups are now under the pod cgroups. The --total-memory is set to 256MiB. The Pod was killed after it used more than 256MiB memory. The node was not killed. The issue was fixed.\nConclusion So the issue was a combination of old gVisor version and incorrect configuration. The gVisor version on the production cluster was built before the fix was merged. The configuration was missing the systemd-cgroup option. The issue was fixed by upgrading the runsc binary and adding the systemd-cgroup option to the configuration.\n","date":"2024-04-15T23:38:00+08:00","permalink":"https://charlie0129.github.io/blog/p/gvisor-oom-debug/","title":"gVisor Killed My Node - A Debug Process"},{"content":"Introduction Re-encoded BD anime series are my go-to when watching animes. They have great quality while being a fraction of the size of a Blu-ray disc. Let alone they come with all kinds of extra contents.\nNowadays, re-encoded BD anime series usually have video tracks compressed using 10-bit x265 (a software implementation of the H.265/HEVC video compression standard), and audio tracks compressed using FLAC (a lossless audio codec).\nThe video tracks are great, generally a good balance between size and quality, no complaints. The audio tracks are my main problem. Sometimes they take up too much space with no real benefit for most viewers.\nOther versions like WEB-DL are not considered because they are already heavily compressed so they don\u0026rsquo;t have the space concerns.\nVideo Tracks I am totally satisfied with the video tracks, because:\nthe x265 encoder is very efficient (especially at variable bitrate controlling, which most hardware encoders still suffer from, i.e. NVIDIA NVENC, Intel QuickSync, and AMD VCN), being able to compress video to a fraction of the size of the BD original while preserving video quality; the encoding team usually does a great job doing manual optimizations to fix problems in the original video, such as aliasing, ringing, and color banding. (Some team even goes as far as AI upscaling the video, although a bit controversial.) Despite being one third (or even less) of the original bitrate, you can barely tell the difference between the re-encoded video and the original one, even when comparing side-by-side. In fact, the re-encoded video is sometimes even better due to the manual optimizations by the encoding team. This saves a lot of space while preserving video quality.\nVideo encoding is a complex topic, involving a huge amount of trial and error, even x265-encoding alone has a TON of dials and knobs to tweak, and this is only part of the whole encoding process. I will not go into details here, simply because of my lack of knowledge.\nSamples For example, you can see what the encoding team (VCB-Studio) does to optimize the original video quality from the note of Haiyore! Nyaruko-san / 潜行吧！奈亚子 :\nFor Season 1, the source is of nothing special. Its native resolution is 720p, then upscaled by a poor algorithm. This leads to many defects: lines featuring severe aliasing and ringing as well as serious ringing on the image borders. Dark scenes are widely spread throughout Season 1, together with the plentiful use of dark fades, resulting in heavy colour banding. For the lines, most of the problems can be solved by descaling and reconstruction, then slight AA and moderate de-ringing as supplements. For colour banding, since most scenes have only a little colour banding, in order to deal with severe colour banding in dark scenes, we designed a complex mask for protection in combination with luminance information. We also use a colour banding detection algorithm. Thanks to all those, we finally adaptively controlled the strength of de-banding. At the end, we added some grains to improve the viewing exprience.\n第一季原盘画质一般。原生分辨率为 720p，被比较劣质的算法拉升上来，导致线条带有严重锯齿和振铃，而且画面的边框部分有很强的拉升带来的边缘振铃。画面暗场极多，而且大量运用暗式淡入淡出，这些场景出现了严重的色带。 对于线条，大部分问题都可以通过逆向拉升再重构解决，然后补上轻微的抗锯齿处理和中等强度的去振铃处理。对于色带，由于大部分场景的色带都很少，只有少部分暗场有较严重的色带，我们结合亮度信息设计了复杂的保护手段，并使用了一个色带检测算法，根据检测的结果对去色带的力度进行了自适应调整。最后加上了一定强度的动态噪点，以改善原盘噪点的观感。\nMediaInfo: Haiyore! Nyaruko-san episode 1 from VCB-Studio 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 General Unique ID : 276260962046491867529669363000855506406 (0xCFD5ED098E39DF78A31330DA2C4C2DE6) Complete name : /Volumes/pool-raidz-3x4tb-0/public/anime/[VCB-Studio] Haiyore! Nyaruko-san/[VCB-Studio] Haiyore! Nyaruko-san [Ma10p_1080p]/[VCB-Studio] Haiyore! Nyaruko-san [01][Ma10p_1080p][x265_flac].mkv Format : Matroska Format version : Version 4 File size : 946 MiB Duration : 23 min 56 s Overall bit rate mode : Variable Overall bit rate : 5 524 kb/s Encoded date : UTC 2022-05-28 10:09:48 Writing application : mkvmerge v48.0.0 (\u0026#39;Fortress Around Your Heart\u0026#39;) 64-bit Writing library : libebml v1.4.0 + libmatroska v1.6.0 Video ID : 1 Format : HEVC Format/Info : High Efficiency Video Coding Format profile : Main 10@L4.1@High Codec ID : V_MPEGH/ISO/HEVC Duration : 23 min 56 s Bit rate : 4 028 kb/s Width : 1 920 pixels Height : 1 080 pixels Display aspect ratio : 16:9 Frame rate mode : Constant Frame rate : 23.976 (24000/1001) FPS Color space : YUV Chroma subsampling : 4:2:0 Bit depth : 10 bits Bits/(Pixel*Frame) : 0.081 Stream size : 690 MiB (73%) Writing library : x265 3.5+97-ga456c6e73+1-g9859a8cb5:[Windows][clang 14.0.0][64 bit] Kyouko 10bit+8bit+12bit Encoding settings : rc=crf / crf=14.0000 / qcomp=0.65 / qpstep=4 / stats-write=0 / stats-read=0 / vbv-maxrate=38000 / vbv-bufsize=40000 / vbv-init=0.9 / min-vbv-fullness=50.0 / max-vbv-fullness=80.0 / crf-max=0.0 / crf-min=0.0 / no-lossless / no-cu-lossless / aq-mode=3 / aq-strength=1.00 / aq-bias-strength=1.00 / cbqpoffs=-2 / crqpoffs=-2 / ipratio=1.40 / pbratio=1.20 / psy-rd=2.00 / psy-rdoq=1.00 / deblock=-1:-1 / ref=5 / limit-refs=0 / no-limit-modes / bframes=10 / b-adapt=2 / bframe-bias=0 / b-pyramid / b-intra / weightp / weightb / min-keyint=1 / max-keyint=360 / rc-lookahead=80 / gop-lookahead=0 / scenecut=40 / hist-scenecut=0 / radl=0 / max-cu-size=32 / min-cu-size=8 / me=3 / subme=5 / merange=38 / rdoq-level=1 / rd=5 / rdpenalty=0 / dynamic-rd=0.00 / rd-refine / ----- / cutree / no-sao / rect / no-amp / no-open-gop / wpp / no-pmode / no-pme / no-psnr / no-ssim / nr-intra=0 / nr-inter=0 / no-constrained-intra / no-strong-intra-smoothing / max-tu-size=16 / tu-inter-depth=4 / tu-intra-depth=4 / limit-tu=0 / qg-size=32 / qpmax=69 / qpmin=0 / ----- / cpuid=1111039 / frame-threads=4 / numa-pools=+ / log-level=2 / input-csp=1 / input-res=1920x1080 / interlace=0 / level-idc=0 / high-tier=1 / uhd-bd=0 / no-allow-non-conformance / no-repeat-headers / no-aud / no-hrd / info / hash=0 / no-temporal-layers / lookahead-slices=0 / no-splice / no-intra-refresh / no-ssim-rd / signhide / tskip / max-merge=5 / temporal-mvp / no-frame-dup / no-hme / no-analyze-src-pics / no-sao-non-deblock / selective-sao=0 / no-early-skip / no-rskip / no-fast-intra / no-tskip-fast / no-splitrd-skip / zone-count=0 / no-strict-cbr / no-rc-grain / no-const-vbv / sar=0 / overscan=0 / videoformat=5 / range=0 / colorprim=1 / transfer=1 / colormatrix=1 / chromaloc=0 / display-window=0 / cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / hist-threshold=0.03 / no-opt-cu-delta-qp / no-aq-motion / no-hdr10 / no-hdr10-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=0 / analysis-save-reuse-level=0 / analysis-load-reuse-level=0 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=1 / refine-ctu-distortion=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-analysis-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei / no-hevc-aq / no-svt / no-field / qp-adaptation-range=1.00 / scenecut-aware-qp=0conformance-window-offsets / right=0 / bottom=0 / decoder-max-rate=0 / no-vbv-live-multi-pass Default : Yes Forced : No Color range : Limited Color primaries : BT.709 Transfer characteristics : BT.709 Matrix coefficients : BT.709 Audio ID : 2 Format : FLAC Format/Info : Free Lossless Audio Codec Codec ID : A_FLAC Duration : 23 min 56 s Bit rate mode : Variable Bit rate : 1 494 kb/s Channel(s) : 2 channels Channel layout : L R Sampling rate : 48.0 kHz Frame rate : 11.719 FPS (4096 SPF) Bit depth : 24 bits Compression mode : Lossless Stream size : 256 MiB (27%) Writing library : libFLAC 1.3.2 (UTC 2017-01-01) Language : Japanese Default : Yes Forced : No Menu 00:00:00.000 : en:Chapter 01 00:01:14.992 : en:Chapter 02 00:02:45.040 : en:Chapter 03 00:13:08.997 : en:Chapter 04 00:22:10.037 : en:Chapter 05 00:23:40.002 : en:Chapter 06 As of video quality, you can verify yourself. Here are some frames extracted from Haiyore! Nyaruko-san / 潜行吧！奈亚子 (VCB-Studio). Can you tell any difference between the original and the re-encoded video? Remember that the re-encoded video has a bitrate of only 4028 kbps, while the original video will at least triple that.\nThis blog uses Responsive Images to improve experience, so what you see here may be scaled down. To view the original image, you can download the original file from the links provided.\nOriginal Re-encoded original-0 re-encoded-0 original-1 re-encoded-1 original-2 re-encoded-2 original-3 re-encoded-3 Can\u0026rsquo;t notice any difference? Or the re-encoded one look better? Save the image locally and zoom in further :p. Yes, I DO put the correct pictures in the correct places.\nThe static images are already hard to tell apart. The difference will be even smaller when you are watching the video, simply because there are motions in the video and you will not be able to focus on the details.\nSo yeah, the video tracks are great. I will not touch them.\nAudio Tracks Lossless Compression Problem However, the audio part is where I have a problem. It\u0026rsquo;s not about quality, but about size. Let me explain.\nUnlike the video tracks, which is compressed using a lossy codec (H.265/HEVC), the audio tracks are usually losslessly compressed, which means that its quality is 100% same to the original, literally. This is great for archival purposes, but it\u0026rsquo;s a overkill for most people. Because what comes with great quality is size. FLAC tracks usually have a bitrate of 1000+ kbps. One FLAC track in a single episode will typically be over 200 MB in size, as you can seen from the example below. By compressing them to 200 kbps, for example, you save 800+ kbps per track, or put it simply, cut a 200 MB track to 40 MB, which saves a lot of space. This is especially true for animes with multiple FLAC tracks. An extreme example is Kobayashi-san Chi no Maidragon / Miss Kobayachi\u0026rsquo;s Dragon Maid / 小林家的龙女仆 from team AI-Raws. It has three FLAC tracks, with a total bitrate of 3955 (1432+1294+1229) kbps. Compressing them to 600 (200*3) kbps will cut the entire episode size from 1.89 GB to 1.3 GB, which is a 36% reduction in size, saving 400 MB per episode. That\u0026rsquo;s a lot.\nMediaInfo: Kobayashi-san Chi no Maidragon episode 1 from AI-Raws 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 General Unique ID : 39274225615593709220200463866578723060 (0x1D8BF0D2BC4C9D643B9E040209C9E0F4) Complete name : /Volumes/pool-raidz-3x4tb-0/public/anime/[AI-Raws][Miss Kobayachi\u0026#39;s Dragon Maid][BDRip][MKV]/[AI-Raws] 小林さんちのメイドラゴン #01 (BD HEVC 1920x1080 yuv444p10le FLAC 日本語字幕)[6FDC51A9].mkv Format : Matroska Format version : Version 4 File size : 1.89 GiB Duration : 25 min 11 s Overall bit rate mode : Variable Overall bit rate : 10.8 Mb/s Encoded date : UTC 2022-01-24 14:17:11 Writing application : mkvmerge v33.1.0 (\u0026#39;Primrose\u0026#39;) 64-bit Writing library : libebml v1.3.7 + libmatroska v1.5.0 Video ID : 1 Format : HEVC Format/Info : High Efficiency Video Coding Format profile : Format Range@L4@High Codec ID : V_MPEGH/ISO/HEVC Duration : 25 min 11 s Bit rate : 6 700 kb/s Width : 1 920 pixels Height : 1 080 pixels Display aspect ratio : 16:9 Frame rate mode : Constant Frame rate : 23.976 (24000/1001) FPS Chroma subsampling : 4:4:4 Bit depth : 10 bits Bits/(Pixel*Frame) : 0.135 Stream size : 1.18 GiB (62%) Writing library : x265 2.9+8-27d8424c799d:[Windows][MSVC 1900][64 bit] 10bit Encoding settings : cpuid=1111039 / frame-threads=4 / numa-pools=16 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=3 / input-res=1920x1080 / interlace=0 / total-frames=0 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=4 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop / min-keyint=23 / keyint=250 / gop-lookahead=0 / bframes=4 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=25 / lookahead-slices=4 / scenecut=40 / radl=0 / no-intra-refresh / ctu=64 / min-cu-size=8 / rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=2 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / strong-intra-smoothing / max-merge=3 / limit-refs=3 / limit-modes / me=3 / subme=3 / merange=57 / temporal-mvp / weightp / no-weightb / no-analyze-src-pics / deblock=0:0 / sao / no-sao-non-deblock / rd=4 / no-early-skip / rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / no-b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=2.00 / psy-rdoq=1.00 / no-rd-refine / no-lossless / cbqpoffs=6 / crqpoffs=6 / rc=crf / crf=14.5 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=0 / vbv-maxrate=25600 / vbv-bufsize=10240 / vbv-init=0.9 / crf-max=0.0 / crf-min=0.0 / ipratio=1.40 / pbratio=1.30 / aq-mode=1 / aq-strength=1.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=31 / qpmin=0 / no-const-vbv / sar=0 / overscan=0 / videoformat=5 / range=0 / colorprim=1 / transfer=2 / colormatrix=2 / chromaloc=0 / display-window=0 / max-cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / no-hdr / no-hdr-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=5 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-mv-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei Default : Yes Forced : No Color range : Limited Color primaries : BT.709 Audio #1 ID : 2 Format : FLAC Format/Info : Free Lossless Audio Codec Codec ID : A_FLAC Duration : 25 min 11 s Bit rate mode : Variable Bit rate : 1 432 kb/s Channel(s) : 2 channels Channel layout : L R Sampling rate : 48.0 kHz Frame rate : 11.719 FPS (4096 SPF) Bit depth : 24 bits Compression mode : Lossless Stream size : 258 MiB (13%) Writing library : libFLAC 1.2.1 (UTC 2007-09-17) Default : Yes Forced : No Audio #2 ID : 3 Format : FLAC Format/Info : Free Lossless Audio Codec Codec ID : A_FLAC Duration : 25 min 11 s Bit rate mode : Variable Bit rate : 1 294 kb/s Channel(s) : 2 channels Channel layout : L R Sampling rate : 48.0 kHz Frame rate : 11.719 FPS (4096 SPF) Bit depth : 24 bits Compression mode : Lossless Stream size : 233 MiB (12%) Title : キャストコメンタリー Writing library : libFLAC 1.2.1 (UTC 2007-09-17) Default : No Forced : No Audio #3 ID : 4 Format : FLAC Format/Info : Free Lossless Audio Codec Codec ID : A_FLAC Duration : 25 min 11 s Bit rate mode : Variable Bit rate : 1 229 kb/s Channel(s) : 2 channels Channel layout : L R Sampling rate : 48.0 kHz Frame rate : 11.719 FPS (4096 SPF) Bit depth : 24 bits Compression mode : Lossless Stream size : 221 MiB (11%) Title : スタッフコメンタリー Writing library : libFLAC 1.2.1 (UTC 2007-09-17) Default : No Forced : No Text ID : 5 Format : PGS Muxing mode : zlib Codec ID : S_HDMV/PGS Codec ID/Info : Picture based subtitle format used on BDs/HD-DVDs Duration : 24 min 30 s Bit rate : 277 kb/s Count of elements : 2364 Stream size : 48.5 MiB (3%) Language : Japanese Default : Yes Forced : No Lossy Compression But compressing them so hard will certainly bring quality losses\u0026hellip; Or will it? Well, yes, there must be quality loss when using lossy codecs, but the question is how much? Is it acceptable? You have seen that the example video tracks are compressed to 4028 kbps, which is a lot less than the original video. But the video tracks are still great. So, is the quality loss acceptable for audio tracks?\nIn fact, using a modern codec at 256 kbps (or even half that), the quality loss is so little that I can almost guarantee that you will NOT hear a difference for two reasons:\nmost codecs are considered \u0026ldquo;transparent\u0026rdquo; at 256 kbps or above (better codes will be \u0026ldquo;transparent\u0026rdquo; at even lower bitrate), which means that you cannot tell the difference between the original and the compressed one (more on this later); when watching anime, you are usually not paying attention to details in the audio, but the video. This makes the difference in audio tracks even less noticeable. So we can safely compress audio without worrying about quality loss. But how much should we compress? Well, that depends on your own preference. Generally, you can choose a codec and a bitrate that is better than what is \u0026ldquo;transparent\u0026rdquo; to you.\nBut hold on, how do I know what is \u0026ldquo;transparent\u0026rdquo; to me?\nAudio Transparency There is a concept called \u0026ldquo;transparent\u0026rdquo; in audio encoding, which means the compressed audio is so good that a person cannot tell a difference between the original and the compressed one. This is usually done through ABX blind tests, where the listener is not told which one is the original and which one is the compressed one. If the listener cannot tell the difference, then the audio is considered transparent. Of course, this differs from person to person. You have to do the test with your own ears with different encoders at different bitrates to find your \u0026ldquo;transparent\u0026rdquo; threshold (which codec at which bitrate). When you find your threshold, any better codec at any better bitrate will also be transparent to you.\nAbout Bitrate and quality:\nThey cannot be compared directly. For example, 320 kbps MP3 is not the same as 320 kbps AAC. (AAC is generally considered a better codec than MP3, so it can achieve the same quality with a lower bitrate.)\nWhen dealing with the same codec (same encoder, same settings), the higher the bitrate, the better the quality. For example, MP3 at 320 kbps wll have better quality than MP3 at 256 kbps.\nBut when dealing with different codecs, the bitrate is not the only factor that affects the quality. The codec itself, the encoder, encoding settings, and etc. all affect quality. For example, AAC (qaac, vbr, default settings) at 256 kbps is often said to have similar quality to MP3 at 320 kbps.\nBut the general consensus is that Opus is transparent at 160kbps and above, and AAC is transparent at 192kbps and above:\nA blind test of multiple codecs at ~192kbps VBR shows that codecs at 192k is transparent to the test subject. Opus being the most transparent codec at 192k;\nTests of different encoders at different bitrates published by SoundExpert show that most codecs at 128kbps and above are transparent to most test subjects. Detailed results can be found at http://soundexpert.org/encoders-128-kbps;\nA topic from HydrogenAudio forum even shows that Opus at ~80kbps is transparent to an average listener;\nIt\u0026rsquo;s interesting to observe that 4 members have mentioned Opus@80kbps as point where is hard to spot artifacts for them.\nOpus ~80 kbps is roughly equivalent to LAME ~130 kbps (V5) which lands in an \u0026ldquo;excellent\u0026rdquo; area of quality (MOS 4.5+) http://listening-tests.hydrogenaud.io/sebastian/mp3-128-1/results.htm\nSo one could say that Opus 80 kbps is \u0026ldquo;excellent\u0026rdquo; at least for an average listener. It\u0026rsquo;s clear that an experienced listeners can spot artifacts at much higher rates.\nI never noticed any artifacts at 80kbps though, though I haven\u0026rsquo;t tried to find any either. Quote from: noiselab on 2017-09-18 06:49:15\nI tried it and for me it\u0026rsquo;s 80 kbit. Quote from: hlloyge on 2017-09-18 11:49:01\nI\u0026rsquo;ll be honest, I struggled to ABX at 64kbps. 80kbps is enough for me. Quote from: Funkstar De Luxe on 2017-09-18 13:35:01\n80kbps is pretty much my limit as well, can\u0026rsquo;t be bothered listening to killer samples all the time. Quote from: bstrobl on 2017-09-18 14:39:30\nAll iTunes music is delivered in AAC at 256kbps (before the arrival of Apple Music Lossless), which is an indication that most people cannot tell the difference as Apple is a company that cares about quality;\nAs you can see I am only focusing on AAC and Opus (as of lossy codecs), because AAC offers good sound quality and great compatibility while Opus has great sound quality and \u0026ldquo;okay\u0026rdquo; compatibility. Other codecs like MP3 are just not good enough to compete with them, so they will not be my choice later.\nListening Samples Now bring your best audio equipment. Same as before, let\u0026rsquo;s listen to some samples to see if you can tell a difference, or you can do blind ABX tests by yourself to find your \u0026ldquo;transparent\u0026rdquo; threshold.\nAll lossy codecs are encoded using VBR (CBR is not considered because it is not size-efficient) with a quality setting to match the resulting bitrates of different encoders as close as possible.\nEncoder versions 1 2 3 4 opusenc 0.2-3-gf5f571b; libopus 1.3.1 OggEnc v2.88; libvorbis 1.3.6 qaac 2.73; CoreAudioToolbox 7.10.9.0 lame 3.100 Note that if you cannot playback some audio tracks, make sure your browser supports this type of audio or you can download the audio tracks and play them locally. Latest versions of Chromium-based browsers should be fine. Safari might have some issues with Opus and Ogg.\nOrange / オレンジ For the first example, I will use Orange / オレンジ by 7!! \u0026mdash; the 2nd ending song from Shigatsu wa Kimi no Uso / 四月は君の嘘 / Your Lie in April / 四月是你的谎言 for its female voices. The audio is not complex (simpley put, do not contain many instruments), so it is easier for the encoders to achieve good results.\nVocal Reference track:\nflac, best, 893k Compressed tracks:\nVBR Bitrate Opus AAC Ogg MP3 ~256k opusenc, vbr256, 276k qaac, q109, 225k oggenc2, q8, 234k lame, v0, 249k ~192k opusenc, vbr192, 212k qaac, q91, 171k oggenc2, q6, 178k lame, v2, 182k ~160k opusenc, vbr160, 180k qaac, q82, 142k oggenc2, q5, 153k lame, v4, 148k ~128k opusenc, vbr128, 146k qaac, q64, 113k oggenc2, q4, 131k lame, v5, 126k ~96k opusenc, vbr96, 111k qaac, q45, 86k oggenc2, q2, 92k lame, v7, 98k ~64k opusenc, vbr64, 76k qaac, q27, 66k oggenc2, q0, 61k lame, v9, 69k Vocal with simple instruments Reference track:\nflac, best, 920k Compressed tracks:\nVBR Bitrate Opus AAC Ogg MP3 ~256k opusenc, vbr256, 274k qaac, q109, 236k oggenc2, q8, 239k lame, v0, 256k ~192k opusenc, vbr192, 210k qaac, q91, 180k oggenc2, q6, 183k lame, v2, 183k ~160k opusenc, vbr160, 179k qaac, q82, 149k oggenc2, q5, 158k lame, v4, 147k ~128k opusenc, vbr128, 144k qaac, q64, 119k oggenc2, q4, 134k lame, v5, 126k ~96k opusenc, vbr96, 109k qaac, q45, 90k oggenc2, q2, 95k lame, v7, 100k ~64k opusenc, vbr64, 75k qaac, q27, 66k oggenc2, q0, 62k lame, v9, 70k Vocal with more complex instruments Reference track:\nflac, best, 1089k Compressed tracks:\nVBR Bitrate Opus AAC Ogg MP3 ~256k opusenc, vbr256, 262k qaac, q109, 270k oggenc2, q8, 266k lame, v0, 288k ~192k opusenc, vbr192, 200k qaac, q91, 200k oggenc2, q6, 201k lame, v2, 206k ~160k opusenc, vbr160, 169k qaac, q82, 165k oggenc2, q5, 170k lame, v4, 154k ~128k opusenc, vbr128, 136k qaac, q64, 129k oggenc2, q4, 142k lame, v5, 134k ~96k opusenc, vbr96, 102k qaac, q45, 96k oggenc2, q2, 97k lame, v7, 101k ~64k opusenc, vbr64, 69k qaac, q27, 69k oggenc2, q0, 62k lame, v9, 68k Vision The second song will be a more complex, or demanding one \u0026mdash; Vision by 中島岬:\nVision reference track:\nflac, best, 1058k Vision compressed tracks:\nVBR Bitrate Opus AAC Ogg MP3 ~256k opusenc, vbr256, 260k qaac, q109, 297k oggenc2, q8, 290k lame, v0, 288k ~192k opusenc, vbr192, 197k qaac, q91, 218k oggenc2, q6, 212k lame, v2, 209k ~160k opusenc, vbr160, 165k qaac, q82, 178k oggenc2, q5, 175k lame, v4, 156k ~128k opusenc, vbr128, 133k qaac, q64, 142k oggenc2, q4, 139k lame, v5, 135k ~96k opusenc, vbr96, 100k qaac, q45, 109k oggenc2, q2, 99k lame, v7, 109k ~64k opusenc, vbr64, 66k qaac, q27, 77k oggenc2, q0, 63k lame, v9, 73k You can clearly see the AAC and MP3 encoders tend to give a higher bitrate than target bitrate in VBR mode to handle complex songs.\nWant to listen to your own song at differnet bitrates? Use this script to encode it to all formats. This is exactly the same script that I wrote to encode the songs and generate the table above.\nNow, you should have your own understanding of the different codecs: above which bitrate of which codec is transparent to you.\nMy take on the codecs Here is my opinion: since ~192k Opus is already transparent to me (and to most people), why use 1000+ kbps FLAC? I can save a lot of space, without even tell a difference! Unless you have outstandingly gifted ears with high-end equipments, or you do heavy post-processing on audio, you should and will be fine with a transparent lossy encoding.\nAlso, most audio in animes is voice, which is usually not complex for encoders, so the actual bitrate can be even lower to sound good.\nStill writing\u0026hellip; The rest is still being written\u0026hellip;\n","date":"2023-01-17T21:28:00+08:00","permalink":"https://charlie0129.github.io/blog/p/thoughts-on-re-encoded-bd-animes/","title":"Thoughts on Re-Encoded BD Animes"},{"content":"Outline:\nKnow ZFS filesystem and Docker\u0026rsquo;s zfs storage driver How Docker\u0026rsquo;s zfs driver works and Why it is slow Why we can\u0026rsquo;t directly use overlayfs on ZFS Why overlayfs don\u0026rsquo;t accept remote filesystems Why ZFS is identified as a remote filesystem How to actually solve the problem We will dive into the source code of moby, OpenZFS, and Linux kernel to find out.\nNote: I admit this blog is not so beginner-friendly, which requires some prerequisites, otherwise you may have a hard time reading it though. I will give some questions or concepts after each prerequisite to help you know your understanding is enough on this topic.\ngeneral computer/unix concepts (block devices, copy-on-write, mount points) basic filesystem concepts (difference between block devices and filesystems, common filesystems, basic understanding of Linux Virtual Filesystem) the basics of ZFS filesystem (terminologies like datasets and snapshots, what are rollbacks) Docker images (what are image layers, when they are created/deleted, how it works with UnionFS) Background What is ZFS Described as The last word in filesystems, ZFS is scalable, and includes extensive protection against data corruption, support for high storage capacities, efficient data compression, integration of the concepts of filesystem and volume management, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, RAID-Z, native NFSv4 ACLs, and can be very precisely configured. 1\nBy saying ZFS, I am referring to OpenZFS on Linux and FreeBSD: OpenZFS Documentation\nZFS is a great and sophisticated filesystem, really robust and stable. It never failed my expectations. I personally use ZFS on my personal devices, whenever possible, e.g. laptops (Ubuntu Desktop - for its built-in support for ZFS), NAS (TrueNAS SCALE), and servers (Proxmox VE and Ubuntu Server).\nWhat is a Docker storage driver Docker uses storage drivers to store image layers, and to store data in the writable layer of a container. The container’s writable layer does not persist after the container is deleted, but is suitable for storing ephemeral data that is generated at runtime. Storage drivers are optimized for space efficiency, but (depending on the storage driver) write speeds are lower than native file system performance, especially for storage drivers that use a copy-on-write filesystem. Write-intensive applications, such as database storage, are impacted by a performance overhead, particularly if pre-existing data exists in the read-only layer. 2\nBy default, Docker will use overlay2 whenever possible for all Linux distributions.\nZFS and Docker storage driver? The Docker Engine provides a zfs storage drivers on Linux, which requires a ZFS filesystem, allowing for advanced options, such as creating snapshots, but require more maintenance and setup.\nAccording to Docker docs, the zfs storage driver has the following advantages 3 :\nAvoids the container’s writable layer grow too large in write-heavy workloads. Performs better for write-heavy workloads (though not as well as Docker volumes). A good choice for high-density workloads such as PaaS. Hmm, sounds good, right? Well, keep reading. If it is that good, this blog wouldn\u0026rsquo;t exist at the first place.\nIn this blog, zfs refers to Docker\u0026rsquo;s zfs storage driver, mostly, but may also refer to ZFS filesystem. You should be able to distinguish them by context.\nWhat\u0026rsquo;s the problem? There is one single problem with ZFS that bothered me since the very beginning: Docker.\nWhen using Docker on ZFS, one can only use its zfs driver (no, you cannot use overlay2 directly. We will see why later.). Although Docker docs proudly advertise its zfs driver as something high-performance:\nzfs is a good choice for high-density workloads such as PaaS. 3\nBut in practice, this thing is slow as hell, specifically, when creating image layers. Build times can go from a fraction of a second on overlay2 to several minutes on zfs!\nIt is several magnitudes slower, a complete disaster.\nLet\u0026rsquo;s take the Dockerfile in kube-trigger (a project that I worked on recently) as an example.\nClick to see the complete Dockerfile 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 # Copyright 2022 The KubeVela Authors. # # Licensed under the Apache License, Version 2.0 (the \u0026#34;License\u0026#34;); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an \u0026#34;AS IS\u0026#34; BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Note: the ${BIN} needs to be replaced with the actual binary, # otherwise it won\u0026#39;t work. Refer to Makefile for how it can be done. ARG BUILD_IMAGE=golang:1.17 ARG BASE_IMAGE=gcr.io/distroless/static:nonroot # Force native build platform, and cross-build to target platform later. FROM --platform=${BUILDPLATFORM:-linux/amd64} ${BUILD_IMAGE} as builder WORKDIR /workspace COPY go.mod go.mod COPY go.sum go.sum ARG GOPROXY ENV GOPROXY=${GOPROXY} RUN go mod download COPY build build COPY hack hack COPY cmd cmd COPY api api COPY controllers controllers COPY pkg pkg ARG TARGETARCH ARG ARCH ARG TARGETOS ARG OS ARG VERSION ARG GOFLAGS ARG DIRTY_BUILD ARG ENTRY RUN ARCH=${TARGETARCH:-${ARCH:-amd64}} \\ OS=${TARGETOS:-${OS:-linux}} \\ OUTPUT=${BIN} \\ VERSION=${VERSION} \\ GOFLAGS=${GOFLAGS} \\ /bin/sh build/build.sh \\ ${ENTRY} FROM ${BASE_IMAGE} WORKDIR / COPY --from=builder /workspace/${BIN} . ENTRYPOINT [\u0026#34;/${BIN}\u0026#34;] We will focus on L39-L46:\n1 2 3 4 5 6 7 8 ARG TARGETARCH ARG ARCH ARG TARGETOS ARG OS ARG VERSION ARG GOFLAGS ARG DIRTY_BUILD ARG ENTRY You might be thinking, this is just some build args, so what? Yes, this part almost does nothing (creates some image layers), and should finish immediately. That\u0026rsquo;s exactly the case on overlay2, but not on zfs, which will take minutes!\nSuch slow build times are driving me crazy.\nWhy zfs driver is so slow? How ZFS storage driver works? When using docker on a zfs dataset, the only option is Docker\u0026rsquo;s zfs driver, which uses ZFS dataset operations to create layered filesystems. The zfs storage driver for Docker stores each layer of each image as a separate legacy dataset. Even just a handful of images can result in a huge number of layers, each layer corresponding to a legacy ZFS dataset. As a result, there are hundreds of datasets created when only running a dozen containers.\nThe base layer of an image is a ZFS filesystem. Each child layer is a ZFS clone based on a ZFS snapshot of the layer below it. A container is a ZFS clone based on a ZFS Snapshot of the top layer of the image it’s created from. 4\nWhere\u0026rsquo;s the bottleneck? Although when building images it do not have to deal with such many datasets. It will still spend a fair amount of time mounting and unmounting these datasets (can be seen from Docker debug logs).\nWe can take a look at the code from Docker daemon (moby/moby).\nMount will happen (if necessary) whenever Get is called:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 // File: daemon/graphdriver/zfs/zfs.go // Link: https://github.com/moby/moby/blob/7e44b7cddd43b1771a44a2dd56548627e491c950/daemon/graphdriver/zfs/zfs.go#L365-L408 // Get returns the mountpoint for the given id after creating the target directories if necessary. func (d *Driver) Get(id, mountLabel string) (_ containerfs.ContainerFS, retErr error) { d.locker.Lock(id) defer d.locker.Unlock(id) mountpoint := d.mountPath(id) if count := d.ctr.Increment(mountpoint); count \u0026gt; 1 { return containerfs.NewLocalContainerFS(mountpoint), nil } defer func() { if retErr != nil { if c := d.ctr.Decrement(mountpoint); c \u0026lt;= 0 { if mntErr := unix.Unmount(mountpoint, 0); mntErr != nil { logrus.WithField(\u0026#34;storage-driver\u0026#34;, \u0026#34;zfs\u0026#34;).Errorf(\u0026#34;Error unmounting %v: %v\u0026#34;, mountpoint, mntErr) } if rmErr := unix.Rmdir(mountpoint); rmErr != nil \u0026amp;\u0026amp; !os.IsNotExist(rmErr) { logrus.WithField(\u0026#34;storage-driver\u0026#34;, \u0026#34;zfs\u0026#34;).Debugf(\u0026#34;Failed to remove %s: %v\u0026#34;, id, rmErr) } } } }() filesystem := d.zfsPath(id) options := label.FormatMountLabel(\u0026#34;\u0026#34;, mountLabel) logrus.WithField(\u0026#34;storage-driver\u0026#34;, \u0026#34;zfs\u0026#34;).Debugf(`mount(\u0026#34;%s\u0026#34;, \u0026#34;%s\u0026#34;, \u0026#34;%s\u0026#34;)`, filesystem, mountpoint, options) root := d.idMap.RootPair() // Create the target directories if they don\u0026#39;t exist if err := idtools.MkdirAllAndChown(mountpoint, 0755, root); err != nil { return nil, err } if err := mount.Mount(filesystem, mountpoint, \u0026#34;zfs\u0026#34;, options); err != nil { return nil, errors.Wrap(err, \u0026#34;error creating zfs mount\u0026#34;) } // this could be our first mount after creation of the filesystem, and the root dir may still have root // permissions instead of the remapped root uid:gid (if user namespaces are enabled): if err := root.Chown(mountpoint); err != nil { return nil, fmt.Errorf(\u0026#34;error modifying zfs mountpoint (%s) directory ownership: %v\u0026#34;, mountpoint, err) } return containerfs.NewLocalContainerFS(mountpoint), nil } Unmount will happen whenever Put is called:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 // File: daemon/graphdriver/zfs/zfs.go // Link: https://github.com/moby/moby/blob/7e44b7cddd43b1771a44a2dd56548627e491c950/daemon/graphdriver/zfs/zfs.go#L410-L431 // Put removes the existing mountpoint for the given id if it exists. func (d *Driver) Put(id string) error { d.locker.Lock(id) defer d.locker.Unlock(id) mountpoint := d.mountPath(id) if count := d.ctr.Decrement(mountpoint); count \u0026gt; 0 { return nil } logger := logrus.WithField(\u0026#34;storage-driver\u0026#34;, \u0026#34;zfs\u0026#34;) logger.Debugf(`unmount(\u0026#34;%s\u0026#34;)`, mountpoint) if err := unix.Unmount(mountpoint, unix.MNT_DETACH); err != nil { logger.Warnf(\u0026#34;Failed to unmount %s mount %s: %v\u0026#34;, id, mountpoint, err) } if err := unix.Rmdir(mountpoint); err != nil \u0026amp;\u0026amp; !os.IsNotExist(err) { logger.Debugf(\u0026#34;Failed to remove %s mount point %s: %v\u0026#34;, id, mountpoint, err) } return nil } Although Docker will not mount a filesystem twice, changes still exist when consecutive Get/Put call happens.\nI am not an OpenZFS developer, but it seems to me that there is a bottleneck with ZFS with such frequent mount/unmount actions (with a large mount datasets and snapshots).\nAs you can see, Docker is already optimizing this situation by using the mount syscall directly (instead of calling user-space mount command, which will again, after going to kernel-space from user-space, require the kernel to call the zfs mount binary in user-space, due to ZFS\u0026rsquo;s license issues with the vfs_mount in kernel).\nOne possible solution So there is not much to optimize in zfs storage drivers. It is the actual zfs mount process that is slowing image build times down. Now, the problem with Docker\u0026rsquo;s zfs storage driver is clear. There are two options left:\noptimize ZFS mount times\njust get rid of zfs storage driver\nOf course, you can also grab another disk in ext4 as use overlayfs on top of it. But I only have zfs-formatted disks, so I only have the above two options.\nThe first one \u0026ldquo;optimize ZFS mount times\u0026rdquo; isn\u0026rsquo;t really an option. Currently, I don\u0026rsquo;t have the expertise or time to work on OpenZFS.\nWith that out of the way, we only have one option left: \u0026ldquo;do not use zfs storage driver\u0026rdquo;, i.e., use overlay2 on zfs datasets.\nBut it doesn\u0026rsquo;t work Now, the problem is, how do we use overlay2 storage driver on a ZFS filesystem (dataset)?\nSimply put, that\u0026rsquo;s not possible (directly). ZFS makes use of d_revalidate . Having d_revalidate set to not NULL will make overlayfs refuse to work.\nBut why? To understand, we need to analyze some source code from OpenZFS and Linux kernel.\nWhat\u0026rsquo;s d_revalidate? d_revalidate is defined in Linux kernel include/linux/dcache.h:\n1 2 3 4 5 6 // File: include/linux/dcache.h // Link: https://github.com/torvalds/linux/blob/1612c382ffbdf1f673caec76502b1c00e6d35363/include/linux/dcache.h#L128 struct dentry_operations { int (*d_revalidate)(struct dentry *, unsigned int); // The rest are omitted. } ____cacheline_aligned; The kernel documentation has a nice description on d_revalidate.\nTL;DR: d_revalidate is typically used with network filesystems, and is called called when the VFS needs to revalidate a dentry, marking this dentry is still valid or not, to prevent things change without the client being aware of it.\nd_revalidate is called called when the VFS needs to revalidate a dentry. This is called whenever a name look-up finds a dentry in the dcache. Most local filesystems leave this as NULL, because all their dentries in the dcache are valid. Network filesystems are different since things can change on the server without the client necessarily being aware of it.\nThis function should return a positive value if the dentry is still valid, and zero or a negative error code if it isn’t.\nd_revalidate may be called in rcu-walk mode (flags \u0026amp; LOOKUP_RCU). If in rcu-walk mode, the filesystem must revalidate the dentry without blocking or storing to the dentry, d_parent and d_inode should not be used without care (because they can change and, in d_inode case, even become NULL under us).\nIf a situation is encountered that rcu-walk cannot handle, return -ECHILD and it will be called again in ref-walk mode.\nExcerpt from: Overview of the Linux Virtual File System — The Linux Kernel documentation\nWhy use d_revalidate? ZFS makes use of d_revalidate to invalidate dcache after rolling back.\nSee? This is the same situation as the kernel doc describes. When a roll back happens, the underlying files are changed, but the dcache it not updated, so we need to use d_revalidate to mark it as invalid.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 // File: module/os/linux/zfs/zpl_inode.c // Link: https://github.com/openzfs/zfs/blob/1d3ba0bf01020f5459b1c28db3979129088924c0/module/os/linux/zfs/zpl_inode.c#L701-L739 static int #ifdef HAVE_D_REVALIDATE_NAMEIDATA zpl_revalidate(struct dentry *dentry, struct nameidata *nd) { unsigned int flags = (nd ? nd-\u0026gt;flags : 0); #else zpl_revalidate(struct dentry *dentry, unsigned int flags) { #endif /* HAVE_D_REVALIDATE_NAMEIDATA */ /* CSTYLED */ zfsvfs_t *zfsvfs = dentry-\u0026gt;d_sb-\u0026gt;s_fs_info; int error; if (flags \u0026amp; LOOKUP_RCU) return (-ECHILD); /* * After a rollback negative dentries created before the rollback * time must be invalidated. Otherwise they can obscure files which * are only present in the rolled back dataset. */ if (dentry-\u0026gt;d_inode == NULL) { spin_lock(\u0026amp;dentry-\u0026gt;d_lock); error = time_before(dentry-\u0026gt;d_time, zfsvfs-\u0026gt;z_rollback_time); spin_unlock(\u0026amp;dentry-\u0026gt;d_lock); if (error) return (0); } /* * The dentry may reference a stale inode if a mounted file system * was rolled back to a point in time where the object didn\u0026#39;t exist. */ if (dentry-\u0026gt;d_inode \u0026amp;\u0026amp; ITOZ(dentry-\u0026gt;d_inode)-\u0026gt;z_is_stale) return (0); return (1); } d_revalidate is set to the zpl_revalidate function that we have seen above.\n1 2 3 4 5 // File: module/os/linux/zfs/zpl_inode.c // Link: https://github.com/openzfs/zfs/blob/1d3ba0bf01020f5459b1c28db3979129088924c0/module/os/linux/zfs/zpl_inode.c#L830-L832 dentry_operations_t zpl_dentry_operations = { .d_revalidate\t= zpl_revalidate, }; But why is having d_revalidate set to not NULL will make overlayfs refuse to work?\nHow overlayfs refuses d_revalidate-enabled fs To understand how and why How overlayfs refuses d_revalidate-enabled fs, let\u0026rsquo;s turn our focus to the Linux kernel.\nDCACHE_OP_REVALIDATE flag is set If a dentry has d_revalidate set to not NULL, which is the case with ZFS, kernel will mark DCACHE_OP_REVALIDATE in its d_flags. A d_flags is just flags to tell what operations that this dentry supports, and DCACHE_OP_REVALIDATE means it supports d_revalidate operations.\nNow in our case, ZFS uses d_revalidate, so our d_flags have a DCACHE_OP_REVALIDATE present.\nKeep this in mind. This flag will cause overlayfs to identify it as a remote fs. You will see why later.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 // File: fs/dcache.c // Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/dcache.c#L1915-L1943 void d_set_d_op(struct dentry *dentry, const struct dentry_operations *op) { WARN_ON_ONCE(dentry-\u0026gt;d_op); WARN_ON_ONCE(dentry-\u0026gt;d_flags \u0026amp; (DCACHE_OP_HASH\t| DCACHE_OP_COMPARE\t| DCACHE_OP_REVALIDATE\t| DCACHE_OP_WEAK_REVALIDATE\t| DCACHE_OP_DELETE\t| DCACHE_OP_REAL)); dentry-\u0026gt;d_op = op; if (!op) return; if (op-\u0026gt;d_hash) dentry-\u0026gt;d_flags |= DCACHE_OP_HASH; if (op-\u0026gt;d_compare) dentry-\u0026gt;d_flags |= DCACHE_OP_COMPARE; if (op-\u0026gt;d_revalidate) dentry-\u0026gt;d_flags |= DCACHE_OP_REVALIDATE; if (op-\u0026gt;d_weak_revalidate) dentry-\u0026gt;d_flags |= DCACHE_OP_WEAK_REVALIDATE; if (op-\u0026gt;d_delete) dentry-\u0026gt;d_flags |= DCACHE_OP_DELETE; if (op-\u0026gt;d_prune) dentry-\u0026gt;d_flags |= DCACHE_OP_PRUNE; if (op-\u0026gt;d_real) dentry-\u0026gt;d_flags |= DCACHE_OP_REAL; } EXPORT_SYMBOL(d_set_d_op); Mounting process of overlayfs To understand how overlayfs rejects d_revalidate enabled fs, we need to look at the code that mounts overlayfs.\nWhen we mount a overlayfs, we call ovl_mount() in the kernel fs/overlayfs.\n1 2 3 4 5 6 7 // File: fs/overlayfs/super.c // Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/overlayfs/super.c#L2158-L2162 static struct dentry *ovl_mount(struct file_system_type *fs_type, int flags, const char *dev_name, void *raw_data) { return mount_nodev(fs_type, flags, raw_data, ovl_fill_super); } ovl_fill_super() will be called to fill the dir (workdir) where overlayfs mounts, which will call ovl_get_workdir.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 // File: fs/overlayfs/super.c // Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/overlayfs/super.c#L2047-L2079 static int ovl_fill_super(struct super_block *sb, void *data, int silent) { // Omitted till L2048 if (ofs-\u0026gt;config.upperdir) { struct super_block *upper_sb; err = -EINVAL; if (!ofs-\u0026gt;config.workdir) { pr_err(\u0026#34;missing \u0026#39;workdir\u0026#39;\\n\u0026#34;); goto out_err; } err = ovl_get_upper(sb, ofs, \u0026amp;layers[0], \u0026amp;upperpath); if (err) goto out_err; upper_sb = ovl_upper_mnt(ofs)-\u0026gt;mnt_sb; if (!ovl_should_sync(ofs)) { ofs-\u0026gt;errseq = errseq_sample(\u0026amp;upper_sb-\u0026gt;s_wb_err); if (errseq_check(\u0026amp;upper_sb-\u0026gt;s_wb_err, ofs-\u0026gt;errseq)) { err = -EIO; pr_err(\u0026#34;Cannot mount volatile when upperdir has an unseen error. Sync upperdir fs to clear state.\\n\u0026#34;); goto out_err; } } err = ovl_get_workdir(sb, ofs, \u0026amp;upperpath); if (err) goto out_err; if (!ofs-\u0026gt;workdir) sb-\u0026gt;s_flags |= SB_RDONLY; sb-\u0026gt;s_stack_depth = upper_sb-\u0026gt;s_stack_depth; sb-\u0026gt;s_time_gran = upper_sb-\u0026gt;s_time_gran; } } Saw something called lowerdir and upperdir? Here\u0026rsquo;s what they means in overlayfs:\nAn overlay filesystem combines two filesystems - an \u0026lsquo;upper\u0026rsquo; filesystem and a \u0026rsquo;lower\u0026rsquo; filesystem. When a name exists in both filesystems, the object in the \u0026lsquo;upper\u0026rsquo; filesystem is visible while the object in the \u0026rsquo;lower\u0026rsquo; filesystem is either hidden or, in the case of directories, merged with the \u0026lsquo;upper\u0026rsquo; object.\nIt would be more correct to refer to an upper and lower \u0026lsquo;directory tree\u0026rsquo; rather than \u0026lsquo;filesystem\u0026rsquo; as it is quite possible for both directory trees to be in the same filesystem and there is no requirement that the root of a filesystem be given for either upper or lower.\nThe lower filesystem can be any filesystem supported by Linux and does not need to be writable. The lower filesystem can even be another overlayfs. The upper filesystem will normally be writable and if it is it must support the creation of trusted.* extended attributes, and must provide valid d_type in readdir responses, so NFS is not suitable.\nA read-only overlay of two read-only filesystems may use any filesystem type.\nExcerpt from Overlay Filesystem — The Linux Kernel documentation\nAnd ovl_get_workdir will get where the workdir is and it call ovl_make_workdir to make the workdir.\n1 2 3 4 5 6 7 8 // File: fs/overlayfs/super.c // Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/overlayfs/super.c#L1513 static int ovl_get_workdir(struct super_block *sb, struct ovl_fs *ofs, struct path *upperpath) { // Omitted till L1513 err = ovl_make_workdir(sb, ofs, \u0026amp;workpath); } ZFS is identified as a remote fs Now the good bit comes, notice the comments. This where the overlayfs rejects remote fs. (We will see why ZFS is identified as remote fs later.)\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 // File: fs/overlayfs/super.c // Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/overlayfs/super.c#L1434-L1444 static int ovl_make_workdir(struct super_block *sb, struct ovl_fs *ofs, struct path *workpath) { // Omitted till L1433 /* * We allowed sub-optimal upper fs configuration and don\u0026#39;t want to break * users over kernel upgrade, but we never allowed remote upper fs, so * we can enforce strict requirements for remote upper fs. */ if (ovl_dentry_remote(ofs-\u0026gt;workdir) \u0026amp;\u0026amp; (!d_type || !rename_whiteout || ofs-\u0026gt;noxattr)) { pr_err(\u0026#34;upper fs missing required features.\\n\u0026#34;); err = -EINVAL; goto out; } } In ovl_dentry_remote, it directly marks dentry which has DCACHE_OP_REVALIDATE flags (Remember what we said before? ZFS sets this flag.) as remote, and thus the code above will go into the if-condition, then rejecting it.\n1 2 3 4 5 6 7 // File: fs/overlayfs/super.c // Link: https://github.com/torvalds/linux/blob/3bc1bc0b59d04e997db25b84babf459ca1cd80b7/fs/overlayfs/util.c#L97-L101 bool ovl_dentry_remote(struct dentry *dentry) { return dentry-\u0026gt;d_flags \u0026amp; (DCACHE_OP_REVALIDATE | DCACHE_OP_WEAK_REVALIDATE); } Now everything comes together. Ah, this is why having d_revalidate set to not NULL will lead to Linux treating ZFS as a remote filesystem (like NFS) and thus things like overlayfs won\u0026rsquo;t work with ZFS.\nThere is PRs in OpenZFS to fix this problem: https://github.com/openzfs/zfs/pull/9600 , https://github.com/openzfs/zfs/pull/9414 . But currently they are held and I don\u0026rsquo;t the expertise or time to work on it either. Hopefully wish I can pick up that PR and fix it (if possible).\nFinal solution So, the only option left is not possible now. Is there something we can do?\nAs I said earlier, \u0026ldquo;Simply put, that\u0026rsquo;s not possible (directly)\u0026rdquo;. Well, it turns out, there is still an indirect way \u0026ndash; ZFS Volumes. Let\u0026rsquo;s see what Oracle says:\nA ZFS volume is a dataset that represents a block device.\nExcerpt from: https://docs.oracle.com/cd/E19253-01/819-5461/gaypf/index.html\nNote that it is a block device. This is really important, which means we can treat it like a conventional hard drive and do whatever we want on ZFS datasets!\nSince it is a block device, we can use it as a Swap device, iSCSI target, and in this case, a block device holding a ext4 partitation to put overlayfs on.\nSolve problem Finally! We now decide to use ZFS Volumes (zvol) to hold our overlayfs , i.e., overlayfs on top of ext4 on top of zvol on top of ZFS datasets. (Well, it is a bit complex. But trust me, even with so many fs layers, the performance is still wayyyy higher than Docker\u0026rsquo;s zfs driver.)\nLet\u0026rsquo;s fix this now.\nStop Docker:\n1 sudo systemctl stop docker Destroy the dataset that Docker uses previously. You can use zfs list to find all datasets. In our case, it is rpool/ROOT/ubuntu_uzcb39/var/lib/docker.\n1 2 3 # I mount this dataset at /var/lib/docker, which docker uses. Remove it. sudo zfs destroy rpool/ROOT/ubuntu_uzcb39/var/lib/docker -R -r # Be careful! This will destroy datasets recursively. Create a ZFS Volume.\n1 2 3 4 5 sudo zfs create -sV 64G rpool/ROOT/ubuntu_uzcb39/var/lib/docker # rpool/ROOT/ubuntu_uzcb39/var/lib/docker is where the dataset is. Note that this will not be mounted to /var/lib/docker, which is different from the one above. # -V creates a zvol # -s makes it sparse, i.e, dynamically expands instead of taking all defined space # set a max size of 64G Format the zvol. ZFS Volumes are identified as devices in the /dev/zvol/{dsk,rdsk}/pool directory. Since we created a block device, let\u0026rsquo;s format it to ext4.\n1 2 # The zvol we just created is mounted at /dev/zvol/rpool/ROOT/ubuntu_uzcb39/var/lib/docker. sudo mkfs.ext4 /dev/zvol/rpool/ROOT/ubuntu_uzcb39/var/lib/docker Mount the ext4 partitation to /var/lib/docker.\n1 2 3 sudo mkdir -p /var/lib/docker # Mount ext4-formatted zvol (block device) to /var/lib/docker sudo mount /dev/zvol/rpool/ROOT/ubuntu_uzcb39/var/lib/docker /var/lib/docker Check if it is successfully mounted.\n1 2 3 df -hT # /dev/zd0 ext4 63G 5.8G 54G 10% /var/lib/docker # It is mounted, great! Start Docker back up and check status.\n1 2 3 4 5 6 7 8 sudo systemctl start docker docker info # We are using overlay2 now! # \u0026gt; Storage Driver: overlay2 # \u0026gt; Backing Filesystem: extfs # \u0026gt; Supports d_type: true # \u0026gt; Native Overlay Diff: true # \u0026gt; userxattr: false Make changes persistent. Make sure the zvol is automatically mounted to /var/lib/docker after system reboots.\n1 2 3 sudo vim /etc/fstab # Append this: # /dev/zvol/rpool/ROOT/ubuntu_uzcb39/var/lib/docker\t/var/lib/docker\text4\tdefaults\t0\t0 Horrey! Build time are several orders of magnitude faster now!\nZFS - Debian Wiki\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nAbout storage drivers | Docker Documentation\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nDocker storage drivers | Docker Documentation\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nUse the ZFS storage driver | Docker Documentation\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","date":"2022-08-07T08:54:00+08:00","permalink":"https://charlie0129.github.io/blog/p/docker-and-zfs-a-tough-pair/","title":"Docker and ZFS - A Tough Pair"}]