In Minimal Alpine Linux on a 1 GB Btrfs Root Disk I wrote down the recipe I use to build a tiny Alpine instance: boot the ISO in a VM, run setup-alpine with disk=none, partition by hand, setup-disk -m sys /mnt, reboot, then qemu-img convert the disk. It works, and I still use that post as the explanation of why the pieces are the way they are.

What I got tired of is the process. Every image costs a VM, a console session, and a sequence of interactive answers, and none of it is reproducible: if I want the same image with ext4 instead of btrfs, I do the whole thing again and hope I remember every step.

So I replaced it with a script. It builds the disk image as a file, on any Linux host, with no VM and no prompts. It also fixes the two things I did not like about the old recipe: the image now boots under both BIOS and UEFI, and every choice is configurable.

The bundle for this post contains everything:

• mkalpine.sh — the builder
• config.example.sh — every knob with comments
• testboot.sh — boot the result under QEMU and check it came up
• testmatrix.sh — build and boot every supported combination
• hooks/ — the customizations, one file each

1

sudo ./mkalpine.sh -f alpine.img

That is the whole interface. About a minute later there is a 512 MiB sparse image, 102 MiB of it actually allocated, that boots on either firmware. The image is deliberately small — 70-growroot expands the root filesystem to fill whatever disk it lands on, so IMAGE_SIZE only has to hold the build.

Why No VM Is Needed

The manual recipe suggests that installing Alpine requires a running Alpine. It does not. Once you look at what the installer actually does, almost all of it is file manipulation:

1. Unpack a root filesystem.
2. apk add a kernel, an initramfs generator, and a bootloader.
3. Write /etc/fstab, /etc/inittab, and the network config.
4. Run mkinitfs and grub-install.
5. Enable some OpenRC services, which is ln -s in a directory.

Only steps 2 and 4 need to execute Alpine binaries, and a chroot is enough for that. There is no step that needs a booted kernel, a real block device, or firmware.

Alpine’s own setup-disk cannot be reused here, unfortunately. It sources libalpine.sh, calls lbu package, and assumes throughout that it is running on a booted live system. So mkalpine.sh reimplements the same steps directly, following the structure of upstream alpine-make-vm-image and the bootloader logic from setup-disk.

The pipeline is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

truncate -s 512M image      create a sparse file
sfdisk                      partition it
losetup -P                  get /dev/loopNp1..p3
mkfs.vfat / mkfs.btrfs      make filesystems
mount                       mount root, then /boot inside it
tar -x minirootfs           unpack the base system
chroot + apk add            kernel, mkinitfs, grub, fs tools
mkinitfs / grub-install     bootloader for both firmwares
hooks/                      customizations
fstrim; umount; losetup -d  compact and release
qemu-img / zstd / gzip      optional output formats

QEMU is not needed to build. It is used only for qcow2 output and by the boot tests.

One line of that pipeline is less innocent than it looks. losetup -P asks the kernel to scan the partition table, but the /dev/loopNp* nodes are created by udev, after losetup has already exited, so they have to be waited for — and on a host with no udev at all, such as a container whose /dev is a plain tmpfs, they have to be created by hand from the device numbers the kernel publishes in /sys/block/loopN/loopNpM/dev. Both paths are in the script, and the interesting part is the order they are tried in: reaching for mknod early is a mistake, because when udev gets round to the same events it unlinks your node and makes its own, and for the instant in between the path does not exist. On a test host, that instant landed exactly on mkfs.vfat:

1

mkfs.vfat: unable to open /dev/loop0p2: No such file or directory

So the escalation is now slow on purpose — udevadm settle, a second, partx -a, another second, and only then mknod — and once the nodes do appear there is one more settle before anything opens them, so a replacement in flight finishes before mkfs runs rather than during it.

The Disk Layout

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

GPT (default; the protective MBR still carries GRUB's boot.img)

 #  size      type            mount   contents
 1  1 MiB     ef02 BIOS boot  -       grub core.img          (x86_64 only)
 2  64 MiB    ef00 ESP FAT32  /boot   vmlinuz-virt, initramfs-virt,
                                      grub/, EFI/BOOT/BOOTX64.EFI
 3  rest      L    Linux      /       btrfs (default), ext4 or xfs

BIOS : firmware -> MBR boot.img -> p1 core.img -> /boot/grub/grub.cfg
UEFI : firmware -> p2 /EFI/BOOT/BOOTX64.EFI    -> /boot/grub/grub.cfg

Two details are worth calling out.

/boot is the ESP. This is not just to save a partition. It means GRUB only ever has to read FAT. Btrfs, ext4 and XFS roots all work without GRUB parsing them at all, and no on-disk feature the root filesystem gains later can break the bootloader. The old post used Syslinux on ext4 /boot; this is less fragile.

GRUB is installed to the removable path. grub-install --removable --no-nvram writes EFI/BOOT/BOOTX64.EFI (or BOOTAA64.EFI), which is what firmware boots when NVRAM has no entry for the disk. We cannot write the target machine’s NVRAM from a build host anyway, and cloud firmware is generally starting from a blank slate.

On aarch64 the BIOS partition is omitted and only UEFI is set up: Alpine ships grub-efi for arm64 but there is no grub-bios, because the i386-pc target is x86-only.

The MBR Variant

Some providers’ image import still rejects GPT. PARTITION_TABLE=mbr handles that:

1
2
3
4
5
6
7

DOS/MBR

 #  start      type            mount   contents
 -  sector 0   MBR + gap       -       grub boot.img (sector 0) and
                                       core.img (the ~1 MiB gap before p1)
 1  2048       0xEF, bootable  /boot   same as above
 2  after p1   0x83            /       root

No dedicated BIOS boot partition is needed, because GRUB’s i386-pc target embeds core.img in the gap between the MBR and the first partition — which is about 1 MiB, given the 2048-sector start. UEFI still works on most firmware, since the ESP is located by partition type 0xEF. That is a widely followed convention rather than something the spec promises, so MBR mode is “BIOS guaranteed, UEFI very likely”, and GPT stays the default.

What It Trusts

The bootstrap is a single alpine-minirootfs tarball, resolved from latest-releases.yaml for the branch:

• TLS gets the file from the mirror.
• Alpine publishes a .sha256 sidecar next to every release artifact. The build fails and deletes the download on a mismatch.
• Everything after that is apk, with Alpine’s signing keys, from the tarball’s own keyring.

I use the minirootfs rather than a pinned apk.static deliberately: a hardcoded apk.static checksum drifts out of sync with the branch, and the in-image apk always matching the branch matters now that 3.23+ ships apk-tools 3.

ALPINE_BRANCH=latest-stable is the default and gets resolved once, at build time. The repositories written into the image always point at the concrete branch (v3.24), never at latest-stable, so a later apk upgrade on the running machine does not silently jump to the next stable release.

Configuration

Copy the example and edit it; ./config.sh is picked up automatically.

1
2
3

cp config.example.sh config.sh
$EDITOR config.sh
sudo ./mkalpine.sh

Every variable is : "${VAR:=default}" in the script, so the environment works too, which is what I use for one-offs:

1

sudo IMAGE_SIZE=2G ROOT_FS=xfs ./mkalpine.sh out.img

Secrets default to nothing. ROOT_PASSWORD_HASH="" locks the root account, so the failure mode of forgetting to set it is “cannot log in”, not “known root password”, and the example hash ships commented out next to the openssl passwd -6 that generates one. config.sh and the image files are in .gitignore.

If the build host reaches the internet through a proxy, an exported http_proxy / https_proxy / no_proxy is picked up as-is, and BUILD_HTTP_PROXY / BUILD_HTTPS_PROXY / BUILD_NO_PROXY set one for the build alone. It covers both halves of the download: the minirootfs tarball on the host, and apk plus whatever a hook fetches inside the chroot. Both cases of each name are set in the chroot’s environment, because curl reads only the lowercase http_proxy while apk and git take either — and they go into the environment rather than a profile script, so nothing about the proxy survives into the image. A proxy reachable from the build host is usually not reachable from wherever the image is deployed, and the URL frequently carries credentials.

Choosing A Root Filesystem

ROOT_FS takes btrfs, ext4 or xfs, each with its own mkfs and mount option strings:

-K and -E nodiscard skip the discard pass at mkfs time, which is pointless on a sparse file and only produces confusing errors.

Btrfs with zstd stays the default for the same reason as in the old post — on a small disk, transparent compression is worth a lot.

Tuning mkfs For The Storage Underneath

The point of exposing ROOT_MKFS_OPTS is that a cloud disk is rarely a plain disk. config.example.sh carries these examples.

Aligning to a 16 KiB ZFS zvol (volblocksize=16k):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# ext4: the block size cannot exceed the kernel page size (4 KiB on x86_64),
#       so -b 16384 produces a filesystem that will not mount. Align with
#       stride/stripe_width instead: stride = 16K / 4K = 4.
ROOT_MKFS_OPTS="-L alpine-root -m 1 -E nodiscard,stride=4,stripe_width=4"

# or, if you really do want 16 KiB allocation clusters:
ROOT_MKFS_OPTS="-L alpine-root -m 1 -O bigalloc -C 16384"

# xfs: takes the stripe unit directly.
ROOT_MKFS_OPTS="-L alpine-root -d su=16k,sw=1"

# btrfs: nodesize is already 16K; set the sector size explicitly if the host
#        page size differs from the target's.
ROOT_MKFS_OPTS="-L alpine-root -K -s 4096"

Aligning to RAID6 with 6 data disks and a 128 KiB chunk:

1
2
3
4
5
6

# ext4: stride = chunk / block = 128K / 4K = 32
#       stripe_width = stride * data disks = 32 * 4 = 128
ROOT_MKFS_OPTS="-L alpine-root -m 1 -E nodiscard,stride=32,stripe_width=128"

# xfs: su = chunk, sw = number of data disks
ROOT_MKFS_OPTS="-L alpine-root -d su=128k,sw=4"

Tuning Mount Options

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Favour throughput over sync latency. You lose more recent writes on an
# unclean shutdown, which is fine for a rebuildable instance and not fine
# for a database.
ROOT_MOUNT_OPTS="rw,noatime,commit=60,data=writeback"                    # ext4
ROOT_MOUNT_OPTS="rw,noatime,logbsize=256k,allocsize=1m"                  # xfs
ROOT_MOUNT_OPTS="rw,noatime,commit=120,compress=zstd:1,ssd,discard=async,space_cache=v2"

# Favour density on a tiny disk. zstd:6 costs CPU on write; decompression
# stays cheap at any level.
ROOT_MOUNT_OPTS="rw,noatime,compress=zstd:6,ssd,discard=async,space_cache=v2"

# Drop discard=async and rely on the weekly fstrim job instead, if your
# provider's thin pool behaves badly with continuous discards.
ROOT_MOUNT_OPTS="rw,noatime,compress=zstd:3,ssd,space_cache=v2"

These options are used to mount the image during the build as well as being written to /etc/fstab, so a typo fails the build instead of the first boot. There is exactly one deliberate difference between the two, and it is the subject of Compression Needs Forcing At Build Time below.

The Mount Option Gotcha

This one cost me a while, and it is the reason the boot test in the bundle checks mount options rather than trusting them.

ROOT_MOUNT_OPTS in /etc/fstab is not enough. The root filesystem is mounted by the initramfs, long before /etc/fstab exists, and the mount -o remount,rw / that OpenRC does afterwards cannot change every option. Most are remountable, so noatime and commit=60 looked fine. XFS, however, fixes the log buffer size at initial mount:

1
2
3
4
5

# what I asked for
ROOT_MOUNT_OPTS="rw,noatime,logbsize=256k"

# what the running machine actually had
rw,noatime,inode64,logbufs=8,logbsize=32k,noquota

Silently downgraded to the 32k default, with nothing in dmesg about it. Mounting the same image on the build host with the same options honoured logbsize=256k, which made it look like a kernel difference rather than what it was.

The fix is to put the options on the kernel command line too, so that the initial mount is the one I want:

1
2
3

ROOTFLAGS=$(echo "$FSTAB_ROOT_OPTS" | tr ',' '\n' |
	grep -vx -e rw -e ro | tr '\n' ',' | sed 's/,*$//')
[ -n "$ROOTFLAGS" ] && CMDLINE="$CMDLINE rootflags=$ROOTFLAGS"

rw and ro are dropped because the kernel handles those separately and GRUB already passes ro. With Btrfs you end up with two rootflags= on the cmdline, because grub-mkconfig’s 10_linux detects the subvolume and emits its own; ours is appended after GRUB’s and the initramfs takes the last one.

Now:

1

rw,noatime,inode64,logbufs=8,logbsize=256k,noquota

Btrfs Subvolumes

BTRFS_SUBVOL="@" by default. It costs three lines at build time, and retrofitting a root subvolume later means moving every file, so it is worth doing even on a machine too small to keep many snapshots. Compression is also set with btrfs property set, so it holds regardless of mount options. BTRFS_SUBVOL="" gives the top-level layout from the old post. GRUB does not care either way, because /boot is FAT.

Compression Needs Forcing At Build Time

compress=zstd:3 does not compress everything. Btrfs decides per file, from the beginning of the file, whether compressing is worth it — and on ELF binaries it decides wrong. In a default build, 45 MiB out of 83 MiB was stored uncompressed, including libcrypto.so.3 (3.3 MiB), every grub-* tool, busybox and ld-musl. All of those compress to roughly half. btrfs property set does not help here either: a file carrying only the inode flag still goes through the same heuristic.

compress-force skips the decision. The build mounts the root with it, while /etc/fstab and rootflags= keep plain compress=:

1
2

BUILD_ROOT_OPTS=$(printf '%s' "$ROOT_MOUNT_OPTS" |
	sed 's/^compress=/compress-force=/; s/,compress=/,compress-force=/')

That is the one place where what the build mounts deliberately differs from what the image ships. It takes 62 MiB of data on disk down to 56 MiB, the raw image from 108 MiB to 102 MiB, and df on the booted machine from 74.5 MiB used to 67.9 MiB; BTRFS_FORCE_COMPRESS=no restores the old behaviour. The build prints what it achieved —

1

root data 55M on disk for 86M of files (64%)

— because the answer depends on what the image installs, and the failure mode is otherwise silent: the mount options, and findmnt, look exactly the same whether the files got compressed or not.

The heuristic is left alone for the running system on purpose. It exists to avoid burning CPU on data that will not compress, and this image contains about 10 MiB of exactly that: Alpine ships kernel modules as .ko.gz, and xfs.ko.gz, btrfs.ko.gz and friends stay uncompressed whether you force it or not. Forcing only recovers the files the heuristic misjudged.

Running btrfs filesystem defragment -r -czstd over the finished tree is the other way to get here, and it is the worse one. It lands at 58 MiB rather than 56 MiB, the sparse image balloons to 168 MiB before fstrim claws it back, and despite the name it does not defragment anything — compressed extents cap at 128 KiB, so the extent count went up, 1996 → 2082.

One caveat if you ship a compressed artifact rather than the raw image: compressing inside the image makes the outer compressor’s job harder, and the outer compressor is better at it. Forcing shrank the raw image by 5.6% and grew every compressed output — raw.zst 74 → 77 MiB, raw.xz 72 → 76 MiB. If upload size is the thing you actually care about, BTRFS_FORCE_COMPRESS=no is the right setting, and the build says as much when you ask for both.

Sizing /boot

BOOT_SIZE=64M. Measured usage with one kernel is 36 MiB:

1
2
3
4
5
6

13 M   vmlinuz-virt
9.4M   initramfs-virt
8.3M   grub/                (i386-pc and x86_64-efi modules)
6.2M   System.map-6.18.48-0-virt
156K   EFI/BOOT/BOOTX64.EFI
149K   config-6.18.48-0-virt

That leaves headroom for one kernel and no more. Raise it to 128M if you want to keep a second kernel around — linux-lts alongside linux-virt, say — or if you want the previous kernel to survive an apk upgrade so there is something to fall back to. The build prints /boot usage at the end and warns when it lands above 80%.

Output Formats

OUTPUT_FORMATS is a space-separated list; each entry produces one file next to the raw image.

The sizes are from one build of the same default btrfs image, so they are comparable to each other rather than absolute. Note how little the four compressed formats differ — 76 to 81 MiB, under 7% between the best and the worst: the root filesystem is already zstd-compressed, so the outer compressor is mostly squeezing free space. raw.gz is a perfectly reasonable default in exchange for its compatibility.

That is also why BTRFS_FORCE_COMPRESS cuts the other way here. Every one of these numbers except qcow2 is larger than it would be with forcing off — raw.xz most of all, 76 MiB against 72 MiB — because data that btrfs already compressed at zstd:3 in 128 KiB blocks is data xz -9 cannot compress again. Forcing wins on raw and loses on everything else, so the build prints a reminder when you ask for both.

Drop raw from the list to keep only the compressed artifacts. For other hypervisors, convert the raw image yourself:

1
2
3

qemu-img convert -O vmdk out.img out.vmdk     # VMware
qemu-img convert -O vpc  out.img out.vhd      # Hyper-V / Azure
qemu-img convert -O vdi  out.img out.vdi      # VirtualBox

What Must Not Survive Cloning

A disk image is a template that gets cloned N times, so anything unique baked into it stops being unique. This is the part a hand-built VM image usually gets wrong, and it is why I wanted a script in the first place — I am not going to remember all of this at 1am on a provider’s web console.

Filesystem UUIDs do stay identical across clones. Regenerating them on first boot means rewriting fstab and grub.cfg from a running system, which is more fragile than the problem it solves — it only bites if you attach two clones to the same host.

The 80-firstboot hook is the other half of this. It runs in the boot runlevel, before networking and sshd, and generates what was stripped:

1
2

head -c 16 /dev/urandom | od -An -tx1 | tr -d ' \n' >/etc/machine-id
ssh-keygen -A

It also runs after hostname, so the host keys are commented root@alpine rather than root@(none). Cosmetic, but the alternative bothered me.

Finally, the build stamps /etc/image-release with the Alpine version, architecture, build date, builder git commit, and the resolved filesystem and hook configuration. Six months later, working out which build a running VM came from is otherwise guesswork.

Staying Able To Get In

The single most valuable property of a cloud image is that you can get into it when networking is broken, because the provider’s serial console is then your only channel.

• console=ttyS0,115200 on x86_64, console=ttyAMA0,115200 on aarch64, with the getty in /etc/inittab and the tty listed in /etc/securetty. Without the securetty entry, root login on that console is refused and looks exactly like a wrong password.
• GRUB_TERMINAL="console serial" plus GRUB_SERIAL_COMMAND, so the bootloader is reachable over serial too. That is what lets you fix a bad kernel cmdline or boot an older kernel remotely.
• No quiet. Alpine’s setup-disk defaults to KERNELOPTS=quiet; on a machine whose only debugging channel is the serial console, hiding the boot log is the wrong trade.
• GRUB_TIMEOUT=1 rather than 0, so there is a window to interrupt.

grub.cfg is generated by grub-mkconfig rather than hand-written, which matters more than it looks: Alpine’s grub package carries triggers="grub.trigger=/boot", so a later apk upgrade linux-virt regenerates it and the image stays bootable with no intervention. A static config would just be silently overwritten by that same trigger. There is a hand-written fallback for the case where grub-probe cannot cope with the loop device, but I have not needed it.

Hooks

One ordered list, in HOOKS. Remove a name to disable it; drop a file into hooks/ and add its name to extend. Each hook is a standalone sh script run inside the chroot with the configuration exported into its environment.

1

HOOKS="10-network 20-ssh 30-chrony 40-zram 50-logtruncate 60-sysctl 65-cgroups 70-growroot 80-firstboot"

Off by default, because they are opinionated rather than essential: 66-vmmem (guest-side settings so a VM hands freed memory back to the hypervisor), 90-tools (bash, coreutils, iproute2, tcpdump, htop, tmux, vim and friends — about 120 MiB), 91-ufw, 92-sshguard, 93-podman, 94-cloud-init, and 95-dotfiles (git, zsh, lsd and a dotfiles repo, with zsh as root’s login shell).

66-vmmem, 91-ufw, 92-sshguard and 95-dotfiles are cheap enough to add at the default IMAGE_SIZE on any filesystem. The heavier three are where the compression default starts paying for itself: 90-tools + 93-podman + 94-cloud-init together fit on btrfs at 512 MiB — 274 MiB allocated — and run the 381 MiB XFS root out of space partway through installing podman. Raise IMAGE_SIZE if you want them on ext4 or XFS.

A few implementation notes:

20-ssh rewrites existing lines rather than appending. sshd_config takes the first occurrence of a keyword, so appending PermitRootLogin prohibit-password to a file that already contains a commented-out default works, and appending it to one that has an active setting silently does nothing. The hook edits in place.

50-logtruncate truncates rather than renames. Keeping one .0 copy and then truncate -s 0 on the original preserves the inode, so daemons holding the file open keep writing to it. This is the script from the old post, with one change: it prunes /var/log/pods, because kubelet rotates container logs itself and truncating them underneath it loses lines and confuses its rotated-file accounting.

65-cgroups exists because a stock Alpine mounts no cgroups at all. The cgroups service is installed but in no runlevel, so /sys/fs/cgroup is an empty directory until something like k3s or podman mounts its own idea of the right layout. The hook sets rc_cgroup_mode="unified" in /etc/rc.conf and adds the service to the boot runlevel, so everything in default starts with one cgroup v2 hierarchy already there and every controller enabled. OpenRC has defaulted to unified since 0.51, so the setting is belt and braces on a current branch; it is there so the result does not depend on which OpenRC a branch ships, and so CGROUP_MODE=hybrid or legacy is a one-line change for software that still needs v1. This is the whole of the Alpine-specific preparation the k3s docs ask for. The one Kubernetes blocker it does not remove is swap: kubelet refuses to start while swap is on, and 40-zram turns it on, so k3s needs INSTALL_K3S_EXEC="--kubelet-arg=fail-swap-on=false" or an image built without that hook.

66-vmmem is the guest half of Making VMs Give Memory Back to the Host. QEMU and Proxmox already enable virtio-balloon free page reporting, but a stock guest defeats it three ways: it only reports free memory in 2 MiB blocks, so anything freed in smaller pieces stays invisible; THP in always mode does not actually free a partially-released huge page until memory pressure; and page cache is never free, so it is never reported, and an idle guest has no pressure that would evict it. The hook installs a vmmem service in the boot runlevel that sets page_reporting_order (default 2, i.e. 16 KiB, which is what OrbStack uses) and the THP mode (madvise) through sysfs — GRUB is installed before the hooks run, so the cmdline route was not available, and both knobs are read at use rather than at init, so it makes no difference. Page cache is trimmed by a one-minute cron job that writes to the root cgroup’s memory.reclaim: ordinary LRU reclaim of whatever is above VMMEM_CACHE_KEEP MiB, file pages only, so it is not drop_caches and nothing warm goes. That part needs the cgroup v2 tree from 65-cgroups and is the reason the hook insists on CGROUP_MODE=unified unless the trim is turned off. With VMMEM_DAMON_RECLAIM=yes the service also turns on damon_reclaim, which pages out anything untouched for VMMEM_DAMON_MIN_AGE seconds (default 120, the kernel’s) within a CPU budget of VMMEM_DAMON_QUOTA_MS per second; that needs the DAMON-enabled kernel from the next section, and the hook refuses to build without KERNEL_REPO_DIR rather than leave a warning in the boot log. Two things about it are easy to get wrong. Stock damon_reclaim only works while free memory is between 20% and 50% of RAM, which excludes exactly the guest with headroom whose footprint one wants to shrink, so the service raises the upper watermarks to 100% and the selftest checks that it did. And paging out anonymous memory needs somewhere to page it to: 40-zram provides that, so cold heap ends up compressed in the guest at a 3–4:1 ratio and only the saved part is actually reported to the host; without swap DAMON could shed nothing but page cache. The age is a churn-versus-latency trade: it only concerns memory that is still allocated (what a process frees goes back at once either way), so a short value re-pages the idle daemons and shared libraries that a ten-minute cron job touches every cycle, a decompress per page from zram, while a long value keeps an idle daemon’s memory resident for that long. 600–1800 is a sensible middle for a server with periodic jobs. On bare metal, or under a hypervisor with no balloon, all of it is inert; the service says which case it found in the boot log.

70-growroot uses growpart, not sfdisk. Growing a GPT disk also requires relocating the backup header to the new end of the device, and growpart keeps the partition’s start sector untouched, so whatever alignment the image was built with survives. After that it is btrfs filesystem resize max /, resize2fs or xfs_growfs depending on ROOT_FS, then a stamp file and rc-update del growroot default.

91-ufw exploits the fact that ufw only touches netfilter when enabled. All the rules are recorded inside the chroot with ufw itself, and then ENABLED=yes is set in ufw.conf as the last step. No netfilter calls happen during the build.

95-dotfiles is the only hook that needs the network for something other than the Alpine mirror. It shallow-clones DOTFILES_REPO, runs the repo’s own bootstrap.sh, and switches root’s login shell. Two parts of that were more interesting than expected.

Changing the login shell, first: chsh is not in a minimal Alpine — it lives in shadow, which nothing else here wants — so the hook rewrites field 7 of root’s /etc/passwd line directly. Writing the new file elsewhere and cat-ing it back keeps the original inode, mode and owner, which a mv would not:

1
2
3

awk -F: -v OFS=: -v shell="$login_shell" \
	'$1 == "root" { $7 = shell } { print }' /etc/passwd >/tmp/passwd.new
cat /tmp/passwd.new >/etc/passwd

Second, my dotfiles use zsh4humans, which installs itself — plugins, plus prebuilt fzf and gitstatusd binaries — the first time an interactive zsh starts. Doing that during the build instead buys two things: the shell works on a machine with no internet, and a download failure fails the build rather than the first login.

It costs about 67 MiB of files — 26 MiB of packages, 5 MiB of checkout, 28 MiB of z4h cache — which is 21 MiB of actual disk on the compressed btrfs default. 99-selftest checks it the way that matters: it starts an interactive zsh on the booted machine and confirms the repo’s .zshrc really was sourced.

Being the only hook that talks to GitHub also makes it the one most likely to fail a build. Set http_proxy and https_proxy when the path to GitHub is reliably bad rather than occasionally bad.

Weekly fstrim is installed regardless of ROOT_FS. Although a btrfs root already trims itself through discard=async, the job is not all about the root filesystem. A data disk attached to the instance later is quite likely to be ext4 or XFS, and nothing else in the image would ever trim it. For those, periodic trim is the currently recommended approach over -o discard.

The job also cannot be a one-liner, because a minimal Alpine image does not have util-linux:

1
2
3
4
5
6

fstrim -a 2>/dev/null && exit 0

awk '$1 ~ /^\/dev\// && !seen[$2]++ { print $2 }' /proc/mounts |
while IFS= read -r mp; do
	fstrim "$mp" 2>/dev/null
done

fstrim -a walks every mounted filesystem and de-duplicates devices, but that is util-linux’s fstrim. Busybox’s applet takes exactly one mount point and has no -a at all, so on the image as built the -a form fails and the loop does the work. My original fstrim -a || fstrim / looked like it handled that and did not: the fallback trimmed only the root, which is precisely the filesystem that needed it least.

IPv6 For Containers

93-podman has two IPv6 knobs, both off by default. What they do is smaller than it sounds, and the reason is worth writing down, because the failure they prevent is a confusing one: an instance with a working public IPv6 address, a security group that allows the port, and -p 80:80 on a container that answers over IPv4 and is simply not there over IPv6.

Netavark decides port forwarding per address family, from the addresses the container actually has. An IPv4-only network means no ip6 DNAT rules — nft list tables has table ip nat and no table ip6 nat — and podman reserves the host port on 0.0.0.0 alone. There is nothing to connect to, and no error anywhere that says so. The container has no outbound IPv6 either.

So the network has to be dual stack, and for any network you create yourself that is not something an image can decide: the address family is chosen per network, at create time.

1

podman network create --subnet 10.89.0.0/24 --subnet fd42:10:89::/64 services

A ULA and NAT66, not a real prefix, because providers hand the instance one address and delegate nothing — Aliyun’s ENI gets a single DHCPv6 /128, which cannot be split. Netavark masquerades it outbound and DNATs published ports inbound, so the client address is still preserved on the way in. NAT66 is unfashionable; it is also what is on offer.

What the image can do is make the kernel willing to route for such a bridge, which is PODMAN_IPV6=yes and one sysctl:

1

net.ipv6.conf.all.forwarding = 1

And that line is where the interesting part is. Enabling forwarding makes the kernel stop honouring router advertisements — reasonably, since a router is not supposed to take its own configuration from its neighbours. On an image that gets its address and default route from RAs, which is IPV6=slaac, the next boot comes up with neither. accept_ra=2 is the value that means “router that still accepts RAs”:

1
2

net.ipv6.conf.eth0.accept_ra = 2
net.ipv6.conf.default.accept_ra = 2

Per interface, and my first attempt wrote conf/all/accept_ra instead, which is wrong in a way that is invisible unless something checks: unlike forwarding, a write to all is not propagated to interfaces that already exist, and default only seeds interfaces created later. eth0 is created by its driver before the sysctl service runs, so both land somewhere harmless and the interface stays at 1. The image boots, sysctl -a shows all.accept_ra = 2, and the machine has quietly stopped configuring itself from RAs. 99-selftest caught it on the first boot precisely because it reads the value off the interface rather than trusting the file — the check and the bug went in on the same afternoon.

The hook writes those two only when IPV6=slaac, and this is not tidiness. With IPV6=dhcp the address comes from dhcpcd, which processes RAs in userspace and leaves accept_ra at 0; raising it there would have the kernel configure a second address and a second default route alongside dhcpcd’s. The correct value depends on who is listening, so the hook branches on the variable 10-network already used.

Ordering is free here: Alpine’s sysctl service declares before bootmisc and networking declares after bootmisc, so whatever is in /etc/sysctl.d is applied before the interface comes up.

99-selftest checks forwarding is on and — the one that would actually catch the accept_ra trap — that the interface’s accept_ra is not left at 1 while the machine forwards. That is checked as configuration rather than by looking for a live default route, because a network with no IPv6 at all would not have one either, and a false FAIL in the boot report is worse than no check.

The Default Network

PODMAN_IPV6_SUBNET is the second knob: give it a ULA and the default podman network becomes dual stack too, so a bare podman run -p 80:80 behaves like the named-network case above. This is narrower than it sounds — compose projects create their own networks and are unaffected — but it is the case where there is no create command to pass flags to.

It has to be a file, and the reason is the interesting part. The obvious approach does not work:

1
2

# podman network create --subnet 10.88.0.0/16 --ipv6 --subnet fd42:88::/64 podman
Error: network name podman already used: network already exists

The name is always taken, because the default network is not stored anywhere: podman synthesises it from containers.conf on demand. A machine that has created other networks has only those on disk — /etc/containers/networks/ holds a services.json and no podman.json. And the containers.conf side is no help either: there is a default_subnet for the v4 half with no v6 counterpart, and default_subnet_pools is documented as “only used for ipv4 subnets, ipv6 subnets are always assigned randomly”.

So the hook writes the file podman would have synthesised:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

{
  "name": "podman",
  "id": "2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9",
  "driver": "bridge",
  "network_interface": "podman0",
  "subnets": [
    { "subnet": "10.88.0.0/16", "gateway": "10.88.0.1" },
    { "subnet": "fd42:88::/64", "gateway": "fd42:88::1" }
  ],
  "ipv6_enabled": true,
  "internal": false,
  "dns_enabled": true,
  "ipam_options": { "driver": "host-local" }
}

That id is not decoration, and leaving it out is the failure worth knowing about. A config podman cannot parse is skipped:

1

level=warning msg="Network config \"...podman.json\" could not be parsed, skipping: invalid network ID \"\""

One warning line, exit status 0, and the synthesised IPv4-only network still in place — which is to say the knob would appear to work and do nothing. The value above is not arbitrary either: it is sha256("podman"), which is how podman derives the default network’s own id, so the hook generates it with printf podman | sha256sum and the file is byte-identical in effect to what podman had in memory.

I was wrong about this in an earlier draft of this section, which recommended a network create at provisioning time as the same work without the coupling. It is not the same work; it is refused. The coupling to netavark’s on-disk schema is real and the honest mitigation is not avoidance but detection: 99-selftest asks podman whether the default network is dual stack and carries the expected prefix, so a schema change under an upgrade shows up as a FAIL rather than as a warning nobody reads.

Two things none of this does, both of which look like the same bug from a browser. The security group needs IPv6 rules: on Aliyun a rule’s v4 and v6 sources are separate fields, so 0.0.0.0/0 on port 443 does not imply ::/0. And the name has to have an AAAA record.

Writing Your Own

Drop a script in hooks/, add its name to HOOKS. The whole hooks/ directory is copied into the chroot at /tmp/mkalpine, so a hook that needs to install a longer file can keep it in hooks/files/ and copy it from /tmp/mkalpine/files/ rather than embedding it in a heredoc:

1
2
3
4
5

#!/bin/sh
set -eu
apk add --quiet --no-progress my-thing
install -m 0644 /tmp/mkalpine/files/my-thing.conf /etc/my-thing.conf
rc-update add my-thing default

The number prefixes are a label, not a mechanism. This is not run-parts: nothing sorts the directory, and the run order is simply the order of the HOOKS list, so HOOKS="10-network my-thing 80-firstboot" does exactly what it looks like. Your hook can be called anything, go anywhere in the list, and it does not matter that the shipped extras have crowded the 90s — 99-selftest runs last because it is last in the list, not because of its number.

Two things about the environment a hook runs in. Everything in config.sh is exported into it, but nothing else is: the chroot starts from env -i, so a hook sees PATH, HOME, TERM, the config, and a proxy if one is configured. And it has no controlling terminal and /dev/null on stdin, for the reasons in 95-dotfiles above, so a hook cannot stop a build to ask a question — a program that tries gets EOF and fails, which is a build that ends with an error rather than one that waits all night.

A Kernel Of Your Own

Alpine’s kernels are built from one recipe, main/linux-lts in aports, with a config file per flavour. Anything that recipe does not turn on is simply not available, and once in a while that matters: the VM memory post ends with DAMON-based reclaim as the one mechanism that gives a guest a container-like footprint, and no Alpine kernel builds it. Rebuilding the package with a changed config is not hard, but doing it by hand every release is the kind of chore that does not happen. So the builder grew two pieces:

build-kernel.sh rebuilds linux-virt (or linux-lts) in a Docker build, from the aports branch matching the release (by default it asks the mirror what latest-stable is right now, the same way mkalpine.sh does, because the kernel has to come from the release the image uses: the 3.23 recipe still depends on linux-firmware-any, and apk 3 on a 3.24 image answered that by installing every firmware package until the disk ran out), with every kernel/config.d/*.config fragment appended to the flavour’s config. The Dockerfile does the actual work and is written around BuildKit cache mounts: the apk cache (the SDK and the kernel’s build dependencies are about a gigabyte), the distfiles (the kernel tarball and the stable patch), and ccache, so a second build after a config tweak reuses every object the change did not touch. That last one needs compiler_check = content: ccache checks the compiler, and any GCC plugin a compile loads, by mtime and size, and the kernel rebuilds its latent_entropy plugin every build, so with the default every single object missed (5% hits on an hour-long build, measured before I found out why). With it, a rebuild with a different pkgrel takes 11 minutes on the same 8-core box where the cold build took 58, and most of that is linking, module installation and the -dev subpackage, which ccache cannot help with. The signing key is generated once with openssl into ~/.config/mkalpine/ and handed to the build as a secret mount; abuild names the signature after the key file, so the name is part of the repository’s identity and the key is worth keeping. DOCKER_HOST is honoured, which is how I run it: the script sits on the laptop, the build happens on a box with cores, and the finished repository comes back over the same SSH connection.

1
2

DOCKER_HOST=ssh://root@buildbox ./build-kernel.sh
KERNEL_REPO_DIR=kernel-repo sudo ./mkalpine.sh -f alpine.img

Two things in the Dockerfile are there because the first attempt got them wrong. The APKBUILD reads a FLAVOR environment variable, which looks like exactly the knob needed, and is not: it is for adding a flavour, and used on a shipped one it lists the config twice in $source and abuild refuses. The recipe step edits _flavor= and drops the other flavours’ config lines instead. And build args are environment variables inside RUN, so the arg could not be called FLAVOR either. The other is a check after the build: olddefconfig silently drops any symbol whose dependencies are not met, so every CONFIG_X=y in a fragment is grepped out of the final .config and a miss fails the build rather than shipping a kernel without the feature.

KERNEL_REPO_DIR makes mkalpine.sh install from that repository. It is copied into the image under /tmp for the install, listed before the mirror, and the kernel is requested as linux-virt=6.18.55-r99, the exact version read off the package file name. apk records a constraint written that way verbatim in /etc/apk/world, which is the whole pinning mechanism: apk upgrade will move everything else and leave the kernel alone, however many newer stock versions appear. The repository is removed again before the image is sealed; the signing key stays in /etc/apk/keys, so a package from the next build-kernel.sh run installs on the running machine with a plain apk add. The custom package is built with pkgrel=99, which shows up in uname -r as 6.18.55-99-virt and doubles as a visible mark that the kernel is not the stock one. 99-selftest checks all three: the pin is in world, the running kernel matches it, and the temporary repository is gone.

The ongoing cost is honest: pinning the kernel also pins you out of its security updates until you rebuild. The rebuild is one command, and the cache mounts make the second one much faster than the first, but it is a command somebody has to run.

Verifying The Image

An image that builds is not an image that boots, and an image that boots is not an image that is configured the way you asked. So there are two scripts.

Does It Boot

1
2

./testboot.sh -m uefi alpine.img
./testboot.sh -m bios alpine.img

This boots the image headless under QEMU with -snapshot, so the image is never modified, captures the serial console to a file, and waits for a login: prompt. Reaching the login prompt exercises the whole chain in one go: firmware → GRUB → kernel → initramfs → root mount → OpenRC → getty. On failure it prints the last 40 lines of the console, which is usually enough to see where it stopped. -w drops the -snapshot and lets the guest write to the image, for when you want to inspect what a first-boot service did to the disk.

It finds OVMF and AAVMF across the various paths different distributions use, and uses KVM when /dev/kvm is writable. With KVM, BIOS reaches the login prompt in about 16 seconds and UEFI in about 26 — OVMF initialization is the difference.

Is It Configured Correctly

Add the 99-selftest hook and pass -d:

1
2

HOOKS="$HOOKS 99-selftest" sudo ./mkalpine.sh -f alpine.img
./testboot.sh -d alpine.img

The hook installs a service that runs last in the default runlevel and checks the running system against the configuration it was built from. The expectations are baked in at build time, which is the part that makes it useful: a hook that silently did nothing shows up as a failure rather than as a plausible-looking dump.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

===== SELFTEST BEGIN =====
-- identity
  info machine-id=2767df9cefbfd2d2af2d0b8acdfb4aa5
  info hostkey=SHA256:U242JZNiEf+Qo3swIK+MoW4m2bdNb2eCMG5W6VJAwPM (ED25519)
  ok   hostname is alpine
  ok   machine-id is 32 hex digits
  ok   timezone is UTC
-- filesystems
  info root options=rw,noatime,inode64,logbufs=8,logbsize=256k,noquota
  ok   root is xfs
  ok   /boot is vfat
  ok   root mounted with logbsize=256k
  --   btrfs subvolume (not configured)
  ok   fstab references root by UUID
-- boot
  info kernel=6.18.48-0-virt
  info cmdline=BOOT_IMAGE=/vmlinuz-virt root=UUID=e95aee24-... ro
       modules=sd-mod,usb-storage,xfs rootfstype=xfs
       rootflags=noatime,logbsize=256k console=tty0 console=ttyS0,115200
  ok   cmdline is not quiet
  ok   ttyS0 in securetty
-- hooks
  ok   20-ssh: PermitRootLogin is prohibit-password
  ok   20-ssh: listening on 22
  ok   40-zram: swap is ~100% of RAM
  ok   60-sysctl: congestion control is bbr
  ok   65-cgroups: /sys/fs/cgroup is cgroup2
  ok   65-cgroups: memory controller delegated
  ok   70-growroot: removed itself from the default runlevel
  ok   70-growroot: root fs fills the partition
  ok   80-firstboot: host key is not the build host's
  --   91-ufw (disabled)
-- hygiene
  ok   apk cache is empty
  ok   no build scratch left behind
  ok   repositories pinned to v3.24
  ok   repositories do not track latest-stable
-- sizing
  Filesystem                Size      Used Available Use% Mounted on
  /dev/vda3               381.0M    161.2M    219.8M  42% /
  /dev/vda2                63.0M     36.4M     26.6M  58% /boot
SELFTEST RESULT: 59 passed, 0 failed
===== SELFTEST END =====

(abridged; the real report is one line per check)

testboot.sh -d exits non-zero if any check failed, so it works in a loop.

The checks live in hooks/files/selftest.initd as an ordinary shell script rather than a heredoc, and check takes a command with its arguments instead of a string to eval:

1
2
3
4

check "root is $EXPECT_ROOT_FS"       fstype_is / "$EXPECT_ROOT_FS"
check "machine-id is populated"       test -s /etc/machine-id
check "cmdline is not quiet"          not grep -qw quiet /proc/cmdline
check "40-zram: vm.swappiness is 180"   output_is 180 sysctl -n vm.swappiness

Anything needing a pipeline gets a named predicate instead. The first version of this used eval on quoted strings and was unreadable — check "root is btrfs" "awk '\$2 == \"/\" {print \$3}' /proc/mounts | grep -qx btrfs" — which is a good sign that the abstraction was wrong.

Everything At Once

1

sudo ./testmatrix.sh -o /var/tmp/mx

This builds btrfs/ext4/xfs × gpt/mbr, boot-tests all six under both BIOS and UEFI with the selftest enabled, then checks the things that only show up at runtime:

• Growth. Copy an image, truncate -s 4G, and boot it — with testboot.sh -w, so the guest’s writes actually land in the file instead of in QEMU’s -snapshot overlay. Then confirm the root partition grew (911,360 → 8,253,407 sectors) and that it still starts on its original sector, because a growpart that moved the start would silently discard the build-time alignment. Reading the table back from the un-booted copy would have made this test tautological, which is what it was until I looked closely.
• Output formats. Build with all five, confirm each file exists, and boot the qcow2 directly rather than just checking that qemu-img produced something.
• Identity. Boot the same image twice and confirm the two boots produce different SSH host key fingerprints and machine-ids.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

== build and boot matrix ==
ok   build  btrfs/gpt
ok   boot   btrfs/gpt/uefi
ok   boot   btrfs/gpt/bios
...
== growroot on a larger disk ==
ok   grow   4G
ok   grow   root partition 1959936 -> 8253407 sectors, still at 135168
== output formats ==
ok   format formats.img.zst (74M)
ok   boot   qcow2
== identity is per-boot, not per-image ==
ok   hygiene two boots produced different SSH host keys
ok   hygiene two boots produced different machine-ids
== summary ==
29 passed, 0 failed

Four real bugs in this post were found this way. The XFS logbsize downgrade above was one, and the tautological growth check was another. The third was in the selftest itself: I had written

1

check "apk cache is empty" not ls -A /var/cache/apk

which reads correctly and means the opposite, because ls -A exits 0 on an empty directory. A test suite that catches bugs in its own checks is doing its job.

The last one showed up when I dropped the default IMAGE_SIZE from 1 GiB to 512 MiB. All three filesystems still built and booted, but ext4 and XFS started failing 70-growroot: root fs fills the partition, which had been written as “df size is at least 90% of the partition”. Nothing was wrong with the images: df excludes reserved blocks and fixed metadata, and on a 445 MiB root partition that fixed cost is 44 MiB for ext4 and 64 MiB for XFS — 90.0% and 85.6%. The same filesystems on a 957 MiB partition reported 96.5% and 93.3% and passed. A percentage was simply the wrong shape for the check, because the overhead it has to tolerate barely moves with the disk size while the threshold does. The failure it exists to catch is not marginal either — a root that never grew is a 445 MiB filesystem in a 4 GiB partition — so the tolerance is now the larger of 96 MiB and 10%.

Uploading

Most providers want a compressed raw image or a qcow2:

1

sudo OUTPUT_FORMATS="raw.zst" ./mkalpine.sh -f alpine.img

If your provider has no image import at all — which is the common case on the cheap tiers — the restore-over-dd trick from the old post still applies: boot their stock OS, dd the image onto the disk from a rescue environment or over SSH, and reboot. 70-growroot then handles the fact that their disk is larger than the image.

That flow needs a console or a rescue system at one point or another. When the provider offers neither — no VNC, no serial, no rescue mode, just the stock OS and SSH, which is what the smallest NAT’d tiers look like — use the no-console variant in the old post instead: reinstall dd mode arranges a RAM-resident Alpine by rewriting the stock bootloader, then writes the image over the whole disk and reboots, all over SSH. It accepts the raw.gz / raw.zst / raw.xz artifacts from OUTPUT_FORMATS as-is, and it does not modify a Linux image — which, on a headless box, promotes 10-network and 20-ssh from conveniences to the difference between a machine that comes back and a reinstall ticket.

What I Kept From The Old Post

Everything about why. The old post explains the 64 MiB /boot, the compressed Btrfs root, the absence of disk swap, and the individual tweaks in far more detail than a config file comment can, and it is still the reference for doing this by hand on a machine you have already booted. This post is the automation of it.