Networking on Charlie Chiang's blog

Installing Proxmox VE on an OVHcloud Kimsufi KS-5

Mon, 06 Jul 2026 14:30:00 +0800

I ordered an OVHcloud Kimsufi KS-5 dedicated server for 19.90 USD/month. It is not fast by modern standards, but it is a very usable small Proxmox box if you install it carefully.

The goal for this machine was:

Proxmox VE on mirrored ZFS root.
Keep only part of each SSD for the root pool.
Use the remaining SSD space as a fast disposable scratch pool.
Put VMs and CTs behind NAT, because the server only has one usable public IPv4 address and one /128 IPv6 address.
Keep host writes low where it is easy to do so.

This post is written from the actual KS-5 I installed. The finished machine is running Proxmox VE 9.2.

Hardware

The server I received:

Part	Value
Product	Kimsufi KS-5
CPU	Intel Xeon E3-1270 v6, 4C/8T, 3.8 GHz base, 4.2 GHz turbo
RAM	32 GB DDR4 ECC 2400 MHz
Storage	2x Intel P3520 450 GB NVMe SSD
NIC	2x Intel I210 gigabit

The PCI devices looked like this:

1
2
3
4
5
6
7
8


00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers (rev 05)
00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31)
00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)
02:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 02)
03:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 02)
04:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1) (rev 05)
05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

The SSDs were not new in power-on hours, but the wear was low and both drives had no media errors:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# nvme smart-log /dev/nvme0n1
critical_warning : 0
temperature : 23 C
available_spare : 98%
percentage_used : 6%
Data Units Read : 8.41 TB
Data Units Written : 95.99 TB
power_cycles : 107
power_on_hours : 45628
unsafe_shutdowns : 4
media_errors : 0
num_err_log_entries : 0

# nvme smart-log /dev/nvme1n1
critical_warning : 0
temperature : 23 C
available_spare : 98%
percentage_used : 7%
Data Units Read : 142.27 TB
Data Units Written : 112.67 TB
power_cycles : 74
power_on_hours : 54153
unsafe_shutdowns : 5
media_errors : 0
num_err_log_entries : 0

For a cheap dedicated server, that is acceptable.

Why Install Proxmox Manually

OVHcloud can install an OS for you from the control panel. I did not use that path for Proxmox.

The problem is storage layout. Proxmox works best when it owns the disks directly and can put ZFS on the raw devices. The OVHcloud installer tends to build layouts around mdadm RAID and regular filesystems, or it uses ZFS in a way that treats VM/CT storage more like ordinary directories. That leaves some Proxmox storage features unavailable or awkward, especially thin-provisioned disks and the normal ZFS-backed workflow.

So I installed Proxmox myself.

Format NVMe Drives as 4K LBA

Before installing the OS, boot the server into OVHcloud rescue mode and check the NVMe namespace formats.

My Intel P3520 drives supported both 512-byte and 4096-byte LBA formats:

1
2
3
4
5
6
7
8


# nvme id-ns -H /dev/nvme0n1
LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
LBA Format 1 : Metadata Size: 8 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good
LBA Format 2 : Metadata Size: 16 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good
LBA Format 3 : Metadata Size: 0 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best
LBA Format 4 : Metadata Size: 8 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best
LBA Format 5 : Metadata Size: 64 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best
LBA Format 6 : Metadata Size: 128 bytes - Data Size: 4096 bytes - Relative Performance: 0 Best

The drives defaulted to 512-byte LBA. Since these are enterprise SSDs and 4K was available, I reformatted both namespaces to format 3: 4096-byte sectors with no metadata.

This erases all data on the drive.

1
2


nvme format /dev/nvme0n1 -l 3
nvme format /dev/nvme1n1 -l 3

After formatting, nvme list should show 4 KiB + 0 B:

1
2
3


Node Model Namespace Usage Format FW Rev
/dev/nvme0n1 INTEL SSDPE2MX450G7 0x1 450.10 GB / 450.10 GB 4 KiB + 0 B MDV10290
/dev/nvme1n1 INTEL SSDPE2MX450G7 0x1 450.10 GB / 450.10 GB 4 KiB + 0 B MDV10290

Open the IPMI KVM

In the OVHcloud console, open the IPMI KVM.

The KVM launches through an old Java JNLP applet.

One annoying detail: the IP address you use to access the OVHcloud console should be the same public client IP used when opening the IPMI KVM. I hit a failure mode where the KVM was blocked because I accessed them through different egress IPs.

The machine uses an Intel server board. The useful hotkeys are:

F2: BIOS setup.
F6: one-time boot menu.

The boot logo confirms it is an Intel board:

You can mount a virtual ISO from the JViewer client:

Then use F6 and choose the virtual CD-ROM:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


Please select boot device:

UEFI IPv4: Intel I210 Network 00 at Baseboard
UEFI IPv4: Intel I210 Network 00 at Baseboard 2
UEFI IPv6: Intel I210 Network 00 at Baseboard
UEFI IPv6: Intel I210 Network 00 at Baseboard 2
Launch EFI Shell
Enter Setup
UEFI Virtual CDROM 1.00 # Choose this one!
UEFI Misc Device

Do Not Stream the Proxmox ISO Through JViewer

My first attempt was to mount the Proxmox installer ISO directly in JViewer and boot it.

That technically works, but it was unusably slow. Even when I ran JViewer from an OVH server (which should have enough bandwidth), virtual media throughput was capped at around 64 KB/s. Installing Proxmox by streaming a full ISO through that path would take forever.

The fix is to boot a tiny netboot image first.

Boot Proxmox Through netboot.xyz

Download the netboot.xyz UEFI ISO, mount that ISO in JViewer, and boot it. The image is tiny, so the slow virtual media path is no longer a problem.

In netboot.xyz, choose:

1
2
3


Linux Network Installs (64-bit)
Proxmox
Proxmox VE Text Installer

You can choose the debug installer if you want shell access between install stages. That is useful when something goes wrong.

netboot.xyz downloads the Proxmox installer over the server’s own network connection and then boots it. OVHcloud provides DHCP even for dedicated servers, so the installer had network access without manual IP configuration.

One caveat: after netboot.xyz hands off to the Proxmox installer, the installer does not use a serial console, so Serial over LAN will not work. You still need the JViewer window to see and control the installation.

Install Proxmox on ZFS RAID1

In the Proxmox installer, choose ZFS RAID1 across both NVMe drives.

I intentionally did not give the full SSDs to rpool. The installer has an hdsize option in advanced storage options. I set it to 128 GiB so the root pool would be a 2-way mirror, leaving the remaining space on both SSDs unused for a later scratch pool.

This is the important part: decide this during installation. Growing ZFS into extra space is easy. Shrinking an existing ZFS pool is not.

The final root layout on my machine:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


nvme0n1 419.2G disk
|-nvme0n1p1 1000K part
|-nvme0n1p2 1G part vfat
|-nvme0n1p3 128G part zfs_member
`-nvme0n1p4 290.2G part zfs_member

nvme1n1 419.2G disk
|-nvme1n1p1 1000K part
|-nvme1n1p2 1G part vfat
|-nvme1n1p3 128G part zfs_member
`-nvme1n1p4 290.2G part zfs_member

p3 on both drives is the mirrored root pool:

1
2
3
4
5
6
7
8
9


pool: rpool
state: ONLINE
config:

 NAME STATE
 rpool ONLINE
 mirror-0 ONLINE
 nvme-INTEL_SSDPE2MX450G7_CVPF71620037450RGN-part3 ONLINE
 nvme-INTEL_SSDPE2MX450G7_CVPF721600N5450RGN-part3 ONLINE

p4 on both drives is the later scratch pool.

Optional Partition Alignment Fix

The Proxmox installer did not give me a perfectly round 1 MiB-aligned final size for the ZFS partition (nvmeXn1p3). If you care about this, fix it immediately after installation while the layout is still simple.

The safe method is:

Use fdisk /dev/nvmeXn1
List partitions with p and note the start/end sector of part 3.
Delete d part 3
Create a new part 3 with the same start sector and a new end sector. Increase the end sector a bit, align end sector - 1 to 1M (256 4k sectors).
Do not wipe the ZFS signature.
Write changes with w.
Ask ZFS to expand into the recreated partition.

Example:

1

zpool online -e rpool <device>

Note that if you expanded the ZFS part too little (smaller than metaslabs size, typically 1GiB), ZFS may not actually expand the pool to use the empty space. This is fine.

Find the exact <device> with:

1

zpool status -v rpool

This is not required for a working system. I did it because I wanted the 128 GiB root partition to be exact and aligned. So later partition 4 will not be misaligned either.

Fix OVHcloud Boot-to-Disk

After a manual install, the server may not boot straight into Proxmox even though the installation succeeded.

OVHcloud bare-metal boot is not just “BIOS loads local disk”. The normal path is roughly:

The server PXE-boots from the public interface.
OVHcloud DHCP gives the server its public IP and an iPXE loader.
iPXE queries OVHcloud’s internal boot service.
The boot service returns a script based on your configured boot mode.
In boot-to-disk mode, iPXE uses sanboot with an EFI bootloader path.

If you installed through the OVHcloud panel, OVHcloud knows the EFI bootloader path. If you installed manually, that path can be missing or wrong.

When sanboot fails, iPXE falls back to rEFInd. rEFInd scans the EFI system partitions and may choose the wrong bootloader. In my case it picked memtest first:

1
2
3
4


rEFInd - Booting OS

Starting memtest86+x64.efi
Using load options ''

The rEFInd menu showed both memtest and systemd-boot:

1
2
3
4


Boot memtest86+x64.efi from 1021 MiB FAT volume
Boot EFI\systemd\systemd-bootx64.efi from 1021 MiB FAT volume
Boot memtest86+x64.efi from 1021 MiB FAT volume
Boot EFI\systemd\systemd-bootx64.efi from 1021 MiB FAT volume

For Proxmox installed with UEFI and root-on-ZFS, the bootloader is systemd-boot:

1

\EFI\systemd\systemd-bootx64.efi

Set that path through the OVHcloud CLI:

1
2


ovhcloud login
ovhcloud baremetal edit nsXXX.ip-X-X-X.eu --efi-bootloader-path '\EFI\systemd\systemd-bootx64.efi'

After that, boot-to-disk should go directly into Proxmox instead of falling through to rEFInd.

Basic Post-Install ZFS Settings

Enable TRIM on the root pool and set basic dataset properties:

1
2
3
4
5
6


zpool set autotrim=on rpool
zfs set atime=off rpool
# The default 'on' is also lz4. Just set it to lz4 explicitly to be sure.
zfs set compression=lz4 rpool
# 128k is the default (this is good for HDDs). But 16-32k is better for NVMe array and workloads with small files, which is the case for most VMs/CTs and Proxmox OS. I am using 2-way mirror so I don't need to worry about 4k ashift adding up.
zfs set recordsize=16k rpool

recordsize affects file datasets, not zvol block devices. For VM disks backed by ZFS zvols, check the Proxmox storage block size instead. On new Proxmox installations the default is already 16K; older installations may still use 8K. You can check or change it in:

1

Datacenter -> Storage -> local-zfs -> Edit -> Block Size

I also use a few ZFS module options:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# You may remove zfs_arc_max set by the installer if you want to maximize ZFS ARC. Note that ARC may not shrink fast enough under memory pressure, so keep this in mind.

cat >/etc/modprobe.d/zfs.conf <<'EOF'
options zfs zfs_txg_timeout=30
options zfs zfs_trim_txg_batch=128
options zfs zfs_dirty_data_sync_percent=80
options zfs zfs_delay_min_dirty_percent=95
EOF

update-initramfs -u -k all

The intent:

zfs_txg_timeout=30: reduce idle write frequency.
zfs_trim_txg_batch=128: make TRIM batching less tiny for NVMe.
zfs_dirty_data_sync_percent=80 and zfs_delay_min_dirty_percent=95: delay throttling until the dirty-data situation is actually serious.

These are not universal defaults. They fit this small host because the workload is mostly personal VMs/CTs and scratch tasks, not a database with strict latency guarantees.

Reduce Unnecessary Host Writes

I am not using Proxmox clustering on this server, so I disabled the cluster HA services:

1
2
3


systemctl disable --now pve-ha-crm.service
systemctl disable --now pve-ha-lrm.service
systemctl disable --now corosync.service

Limit persistent journald usage:

1
2


sed -i 's/.*SystemMaxUse.*/SystemMaxUse=128M/g' /etc/systemd/journald.conf
systemctl restart systemd-journald

Configure BBR

Use BBR with fq.

On Debian/Proxmox, a later default sysctl file can set net.core.default_qdisc=fq_codel after your file during boot. Use a late filename:

1
2
3
4
5
6


cat >/etc/sysctl.d/99-bbr.conf <<'EOF'
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
EOF

sysctl --system

Check it:

1

sysctl net.core.default_qdisc net.ipv4.tcp_congestion_control

Expected:

1
2


net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

Configure Swap

Always have swap when using ZFS, even with enough RAM. It gives the kernel somewhere to go if ARC does not shrink quickly enough under memory pressure, and it reduces the chance that the OOM killer targets useful applications.

I use ZRAM swap:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


git clone --depth=1 https://github.com/foundObjects/zram-swap.git
cd zram-swap
./install.sh
cd ..
rm -rf zram-swap

cat >/etc/sysctl.d/zram.conf <<'EOF'
vm.swappiness = 180
vm.watermark_boost_factor = 0
vm.watermark_scale_factor = 125
vm.page-cluster = 0
EOF

sysctl --system

You may use disk swap if you want.

My current host has a 38.8 GiB ZRAM swap device:

1
2


NAME TYPE SIZE USED PRIO
/dev/zram0 partition 38.8G 0B 15

SSH and Serial Console

I like SSH sessions to die quickly when the client disappears unexpectedly:

1
2


ClientAliveInterval 60
ClientAliveCountMax 3

Put that in sshd_config or a file under sshd_config.d, then restart SSH.

For emergency access, enable OVHcloud serial-over-LAN by adding a serial console to the Proxmox kernel command line:

1
2
3


additional_cmdline="console=tty0 console=ttyS0,115200n8"
sed -i "s|$| $additional_cmdline|" /etc/kernel/cmdline
proxmox-boot-tool refresh

On my host, /etc/kernel/cmdline now contains:

1

root=ZFS=rpool/ROOT/pve-1 boot=zfs vmlinuz video=vesafb:ywrap,mtrr initrd=initrd.magic console=tty0 console=ttyS0,115200n8

Scratch Storage

The root pool uses 128 GiB on each SSD. The remaining 290.2 GiB on each SSD is free for scratch storage.

I considered two options:

mdadm RAID0 + LVM thin + ext4/XFS.
ZFS striped pool.

The mdadm option may win in some microbenchmarks, but the management cost is higher. I used a ZFS striped pool because it is simple and good enough.

This pool has no redundancy. If either SSD fails, the scratch pool is gone. That is fine for my use case because it is scratch space.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


zpool create -f \
 -o ashift=12 \
 -o autotrim=on \
 scratchpool \
 /dev/disk/by-id/nvme-INTEL_SSDPE2MX450G7_CVPF721600N5450RGN-part4 \
 /dev/disk/by-id/nvme-INTEL_SSDPE2MX450G7_CVPF71620037450RGN-part4

zfs set recordsize=16k scratchpool
zfs set atime=off scratchpool
zfs set compression=lz4 scratchpool
zfs set logbias=throughput scratchpool

zfs create scratchpool/unsafe
zfs set sync=disabled scratchpool/unsafe
zfs set copies=1 scratchpool/unsafe

Current state:

1
2
3
4
5
6
7
8


pool: scratchpool
state: ONLINE
config:

 NAME STATE
 scratchpool ONLINE
 nvme-INTEL_SSDPE2MX450G7_CVPF721600N5450RGN-part4 ONLINE
 nvme-INTEL_SSDPE2MX450G7_CVPF71620037450RGN-part4 ONLINE

Properties:

1
2
3


NAME AVAIL MOUNTPOINT RECSIZE COMPRESS ATIME SYNC COPIES
scratchpool 562G /scratchpool 16K lz4 off standard 1
scratchpool/unsafe 562G /scratchpool/unsafe 16K lz4 off disabled 1

Use scratchpool/unsafe only for data you can recreate. sync=disabled lies to applications about sync writes.

mdadm + LVM Alternative

If you want the non-ZFS scratch layout, create one extra partition on each SSD and build RAID0:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


apt install -y mdadm

mdadm --create /dev/md0 \
 --verbose \
 --level=0 \
 --raid-devices=2 \
 --chunk=128K \
 /dev/disk/by-id/nvme-INTEL_SSDPE2MX450G7_CVPF721600N5450RGN-part4 \
 /dev/disk/by-id/nvme-INTEL_SSDPE2MX450G7_CVPF71620037450RGN-part4

mdadm --detail --scan >> /etc/mdadm/mdadm.conf
update-initramfs -u

For alignment:

mdadm chunk size: 128K. 128K is a good balance between performance and overhead for NVMe drives. If you want to maximize sequential performance, use larger values like 1M. But remember to align filesystem / LVM with this value.
Data disks: 2.
Stripe width: 256K.
LVM data alignment: 1M is fine because it is a multiple of 256K.

NVMe (physical) → mdadm RAID0 → LVM PV → LVM LV / thin pool → filesystem

When you create LVM on top of RAID-0, you must ensure:

LVM PE size = multiple of mdadm chunk size
LVM PV starts aligned to mdadm stripe boundary

Create the LVM layer:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# The default dataalignment value is 1M, which is a multiple of 256K, so it's fine.
pvcreate --dataalignment 1M /dev/md0
# Default VG extent size is often 4M, which is multiples of 1M data alignment, 
# which is also multiples of 256K stripe width. Using the default value is fine.
vgcreate --physicalextentsize 4M vg_scratch /dev/md0
# Thin pool chunk size defines how much physical space is reserved when any 
# part of a chunk is touched. It does not defined write size. So it can be 
# smaller than 256k. The default 64k is fine. I will just match it with the 
# stripe width = 256k.
# The default metadata size can be too big if you are using small driver 
# like 16G Optane drive. Calculate the desired metadata size (bytes)
# by `48 * (total size / chunk size)`.
lvcreate -l 100%FREE --thin vg_scratch/scratch_thin --chunksize 256k --poolmetadatasize 512M

When formatting a filesystem inside a VM or manually on the host, pass the stripe geometry:

1
2
3
4
5
6
7
8
9


# ext4
# stride = mdadm chunk size / filesystem block size = 128k / 4k = 32
# stripe-width = stride * disks = 32 * 2 = 64
mkfs.ext4 -b 4096 -E stride=32,stripe-width=64 /dev/vg_scratch/lv

# XFS
# su = mdadm chunk size = 128k
# sw = data disks = 2
mkfs.xfs -d su=128k,sw=2 /dev/vg_scratch/lv

Fast but unsafe mount options for scratch data:

1
2


ext4: noatime,nodiratime,nobarrier,data=writeback,commit=60
xfs: noatime,nodiratime,logbufs=8,logbsize=256k

For CTs, Proxmox does not make it convenient to set filesystem mount options on a normal CT mountpoint. If you need custom options, create the LV manually, mount it on the host, and bind-mount it into the CT:

1
2
3
4
5
6
7
8


lvcreate -V 16G -T vg_scratch/scratch_thin -n scratch_test
mkfs.xfs -d su=128k,sw=2 /dev/vg_scratch/scratch_test

mkdir -p /mnt/scratch_test
mount -o noatime,nodiratime,logbufs=8,logbsize=256k /dev/vg_scratch/scratch_test /mnt/scratch_test
chmod 777 /mnt/scratch_test

pct set 199 -mp0 /mnt/scratch_test,mp=/mnt/scratch_test

If your workload is random-write heavy and can fit on one SSD, skip RAID0 and LVM entirely. A single raw disk or single partition is often better for small random writes than a layered RAID0 + LVM-thin setup.

Using Scratch Storage in VMs and CTs

For VMs, add a virtual disk on the scratch pool. Enable discard and SSD emulation on the virtual disk. Inside the guest, enable periodic TRIM. When formatting the disk, make sure the filesystem is aligned to the underlying ZFS volsize or LVM stripe geometry. For example, if underlying ZFS volsize is 16K, use mkfs.ext4 -O bigalloc -C 16384 /dev/yourdisk. If the underlying storage is LVM+RAID, use the correct stride and stripe-width options when formatting.

For CTs, add a mountpoint on the scratch pool. ZFS mountpoints are automatically aligned. For LVM+RAID, make sure the filesystem is aligned to the underlying geometry. TRIM is handled by the host for ZFS mountpoints. For LVM+RAID, run pct fstrim on the CT.

To use the scratch pool for temporary data, you can use bind-mounts to mount common scratch directories to scratch disk mountpoints. This avoids changing the config for each applications such as Docker, containerd and etc.

Suppose the scratch disk is mounted at /mnt/unsafe, and you want to use it for Docker, containerd, and ~/.cache. Write the following lines in /etc/fstab:

1
2
3


/mnt/unsafe/docker /var/lib/docker none rbind 0 0
/mnt/unsafe/containerd /var/lib/containerd none rbind 0 0
/mnt/unsafe/.cache /root/.cache none rbind 0 0

1
2
3
4


mkdir -p /mnt/unsafe/{docker,containerd,.cache}
rm -rf /var/lib/docker /var/lib/containerd /root/.cache
mkdir -p /var/lib/docker /var/lib/containerd /root/.cache
mount -av

TRIM

For ZFS pools:

1
2


zpool set autotrim=on rpool
zpool set autotrim=on scratchpool

For VMs, enable discard and SSD emulation on virtual disks, then enable periodic TRIM inside the guest.

For CTs backed by ZFS mountpoints, the host handles it. For CTs backed by other storage such as LVM thin, run pct fstrim:

1
2
3
4
5
6
7
8
9


cat >/usr/local/bin/trimcts.sh <<'EOF'
#!/bin/bash
for id in $(pct list | awk '$2 == "running" {print $1}'); do
 echo "==> Trimming CT $id"
 pct fstrim "$id"
done
EOF

chmod +x /usr/local/bin/trimcts.sh

Then run it from cron or a systemd timer. For example, weekly from cron:

1

0 2 * * 7 /usr/local/bin/trimcts.sh >/var/log/trimcts.log 2>&1

Proxmox Networking on a Single Public IP

Proxmox creates a bridge by default and attaches the physical NIC to it. That is usually the right setup when the server is in your own LAN.

It is not right for this Kimsufi server.

The server has one usable public IPv4 address and one /128 IPv6 address. It does not have a routed public subnet for VMs and CTs. If VMs are bridged directly to the public interface, they do not get usable public addresses.

So I do this instead:

Put the public IPv4 and public IPv6 directly on eno1, not vmbr0. By default, Proxmox puts the public IP on the vmbr0 bridge, which is not what we want. You should move it to the physical interface.
Create vmbr0 as an internal bridge with no physical ports.
Put VMs and CTs on vmbr0.
NAT outbound traffic from vmbr0 to eno1.
Port-forward selected inbound ports to service CTs.

Current /etc/network/interfaces:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
 address <public-ipv4>/24
 gateway <public-ipv4-gateway>

iface eno1 inet6 static
 address <public-ipv6>/128
 gateway <public-ipv6-gateway>

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
 address 10.187.54.1/24
 bridge-ports none
 bridge-stp off
 bridge-fd 0

iface vmbr0 inet6 static
 address fc10:187:54::1/64

source /etc/network/interfaces.d/*

Enable forwarding:

1
2
3
4
5
6


cat >/etc/sysctl.d/99-forward.conf <<'EOF'
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF

sysctl --system

About NAT66: it is usually the wrong IPv6 design. If you have a routed IPv6 prefix, route it to your VM bridge. On this server I only have one /128, so NAT66 is the practical workaround if VMs/CTs need outbound IPv6.

NAT and Port Forwarding

NAT rewrites packet addresses as traffic crosses the host.

For outbound IPv4 NAT, a CT sends a packet like:

1

src=10.187.54.100:51514 dst=1.1.1.1:443

After MASQUERADE on the host:

1

src=<public-ipv4>:51514 dst=1.1.1.1:443

Linux conntrack records that translation so the reply can be translated back and forwarded to the CT.

There is a Proxmox-specific caveat when the firewall is enabled on a VM or CT network interface (firewall=1). Proxmox places that interface behind an fwbr* firewall bridge. With bridge netfilter enabled, conntrack can record the connection while it crosses the firewall bridge, before it reaches the host’s normal NAT path. The later FORWARD rule can accept the packet, but POSTROUTING may reuse the already tracked non-NATed connection instead of attaching MASQUERADE or SNAT. The packet then leaves the public interface with its private source address and never receives a reply.

Put the firewall-bridge tracking in a separate conntrack zone by adding these hooks to the vmbr0 stanza in /etc/network/interfaces:

1
2
3
4


 post-up iptables -t raw -C PREROUTING -i fwbr+ -j CT --zone 1 2>/dev/null || iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1 2>/dev/null || true
 post-up ip6tables -t raw -C PREROUTING -i fwbr+ -j CT --zone 1 2>/dev/null || ip6tables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
 post-down ip6tables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1 2>/dev/null || true

The -C/|| guard matters on a dual-stack vmbr0: ifupdown2 can run the hook once for each address family.

The IPv4 rules cover MASQUERADE/SNAT, while the IPv6 rules are also needed when using NAT66. A characteristic symptom is that NAT works for a guest without the Proxmox interface firewall but fails for an otherwise identical firewall-enabled guest. In conntrack output, the broken flow’s reply destination remains the guest’s private address instead of the host’s public address. Allowing all OUT and FORWARD traffic does not fix it because this is a conntrack/NAT-path problem, not a filter-policy rejection.

Inbound port forwarding is DNAT. A client connects to the public server:

1

src=198.51.100.20:41000 dst=<public-ipv4>:443

The host rewrites the destination before routing:

1

src=198.51.100.20:41000 dst=10.187.54.3:443

I use my own small tool, iptfwd, to manage these rules from a config file: charlie0129/iptfwd.

Example config for this topology:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


defaults:
 public_iface: eno1
 private_iface: vmbr0
 manage_filter: true
 enable_ip_forwarding: true

nat:
 - name: intranet-v4
 source: 10.187.54.0/24
 type: masquerade

 - name: intranet-v6
 source: fc10:187:54::/64
 type: snat
 to_source: <public-ipv6>

rules:
 - name: service-http-v4
 proto: tcp
 public_port: 80
 target: 10.187.54.3
 target_port: 80

 - name: service-https-v4
 proto: tcp
 public_port: 443
 target: 10.187.54.3
 target_port: 443

 - name: service-http-v6
 proto: tcp
 public_port: 80
 target: fc10:187:54::3
 target_port: 80

 - name: service-https-v6
 proto: tcp
 public_port: 443
 target: fc10:187:54::3
 target_port: 443

For IPv4, MASQUERADE is convenient. For NAT66, I prefer explicit SNAT to the public IPv6 address. IPv6 interfaces can have multiple addresses, and explicit SNAT makes the translated source predictable.

If you do not want to use iptfwd, the equivalent basic iptables rules are:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


iptables -t nat -A POSTROUTING -s 10.187.54.0/24 -o eno1 -j MASQUERADE
iptables -A FORWARD -i vmbr0 -o eno1 -j ACCEPT
iptables -A FORWARD -i eno1 -o vmbr0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

public_ipv6=<YOUR_PUBLIC_IPV6>
ip6tables -t nat -A POSTROUTING -s fc10:187:54::/64 -o eno1 -j SNAT --to-source "$public_ipv6"
ip6tables -A FORWARD -i vmbr0 -o eno1 -j ACCEPT
ip6tables -A FORWARD -i eno1 -o vmbr0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

apt install iptables-persistent
netfilter-persistent save

Before setting up DHCP, test NAT manually:

Create a CT on vmbr0.
Set IPv4 to something like 10.187.54.100/24.
Set IPv4 gateway to 10.187.54.1.
Set IPv6 to something like fc10:187:54::100/64.
Set IPv6 gateway to fc10:187:54::1.
Test:

1
2


ping www.google.com
ping6 www.google.com

If that fails, debug NAT before adding DHCP.

DHCP and DNS with Pi-hole

I use Pi-hole as DHCP and DNS for the private bridge because it has a nice web UI and makes static leases easy.

Create a small Alpine CT:

Bridge: vmbr0.
IPv4: 10.187.54.2/24.
IPv4 gateway: 10.187.54.1.
IPv6: fc10:187:54::2/64.
IPv6 gateway: fc10:187:54::1.
CPU: 1 core is enough.
RAM: 128 MB is enough.
Disk: 512 MB is enough.

Install Pi-hole:

1
2
3
4


apk update
apk add curl bash
curl -sSL https://install.pi-hole.net | bash
# You may disable query logs to reduce writes to the database.

If you forget the web password:

1

pihole setpassword

Reduce query-log database writes:

1

pihole-FTL --config database.DBinterval 1800

In the Pi-hole UI:

Enable DHCP for 10.187.54.0/24.
Set the router/gateway to 10.187.54.1.
Configure DNS as desired.
Disable NTP sync under Settings -> All Settings -> Network Time Sync. Uncheck ntp.ipv4/ipv6/sync.active. An unprivileged CT cannot set system time anyway.

Alpine log cleanup I use for small CTs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


cat >/etc/periodic/hourly/logtruncate <<'EOF'
#!/bin/sh

LOG_DIR="/var/log"
MAX_SIZE=$((4 * 1024 * 1024))

find "$LOG_DIR" -type f ! -name '*.0' | while read -r file; do
 size=$(stat -c %s "$file" 2>/dev/null || stat -f %z "$file" 2>/dev/null || echo 0)

 if [ "$size" -gt "$MAX_SIZE" ]; then
 logger -t logrotate "Rotating $file ($size bytes)"
 cp "$file" "$file.0"
 truncate -s 0 "$file"
 logger -t logrotate "Rotated $file"
 fi
done
EOF

chmod +x /etc/periodic/hourly/logtruncate
sed -i 's/^SYSLOGD_OPTS=.*/SYSLOGD_OPTS="-t -s 0"/' /etc/conf.d/syslog

cat >/etc/periodic/daily/apkcacheclean <<'EOF'
#!/bin/sh
find /var/cache/apk -type f -name '*.apk' -mtime +7 -delete
find /var/cache/apk -type f -name '*.tar.gz' -mtime +90 -delete
EOF

chmod +x /etc/periodic/daily/apkcacheclean

Host Monitoring

I keep monitoring tools in Podman containers instead of installing everything directly on the Proxmox host.

Install Podman:

1
2
3
4
5
6


# So Intel PCM can work.
echo msr >> /etc/modules
modprobe msr

# Install podman to run monitoring tools in containers. Do not include any unnecessary packages to keep the host clean and minimal. Remember to install aardvark-dns to make container DNS work.
apt install --no-install-recommends podman aardvark-dns

On current Debian/Proxmox, Podman should use overlay storage on ZFS:

1

podman info --format '{{.Store.GraphDriverName}}'

Expected:

1

overlay

Make podman-restart.service also handle containers with restart-policy=unless-stopped:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


mkdir -p /etc/systemd/system/podman-restart.service.d

cat >/etc/systemd/system/podman-restart.service.d/override.conf <<'EOF'
[Service]
ExecStart=/usr/bin/podman $LOGGING start --all --filter restart-policy=always --filter restart-policy=unless-stopped
ExecStop=/usr/bin/podman $LOGGING stop --all --filter restart-policy=always --filter restart-policy=unless-stopped
EOF

systemctl daemon-reload
systemctl enable podman-restart.service

Enable the Podman socket and Docker-compatible CLI:

1
2
3
4
5


systemctl enable --now podman.socket

apt install --no-install-recommends docker-cli docker-compose
docker context create podman --docker "host=unix:///run/podman/podman.sock"
docker context use podman

An example monitoring compose file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85


services:
 vmagent:
 container_name: vmagent
 image: victoriametrics/vmagent:v1.144.0
 restart: unless-stopped
 mem_limit: 512M
 environment:
 GOMEMLIMIT: 900MiB
 command:
 - -httpListenAddr=0.0.0.0:8428
 - -cacheExpireDuration=5m
 - -promscrape.config=/etc/vmagent/scrape.yml
 - -remoteWrite.url=https://XXX # Your VictoriaMetrics remote write endpoint
 - -remoteWrite.basicAuth.username=XXX
 - -remoteWrite.basicAuth.password=XXX
 - -remoteWrite.maxDiskUsagePerURL=128MiB
 - -remoteWrite.tmpDataPath=/var/vmagent
 - -remoteWrite.vmProtoCompressLevel=3
 - -remoteWrite.flushInterval=66s
 - -remoteWrite.label=instance=XXX
 extra_hosts:
 - "host.docker.internal:host-gateway"
 volumes:
 - ./vmagent/config:/etc/vmagent
 - ./vmagent/data:/var/vmagent

 smartctlexporter:
 container_name: smartctlexporter
 image: prometheuscommunity/smartctl-exporter:v0.14.0
 restart: unless-stopped
 mem_limit: 64M
 user: 0:0
 environment:
 GOMEMLIMIT: 110MiB
 privileged: true

 nodeexporter:
 container_name: nodeexporter
 image: prom/node-exporter:v1.11.1
 restart: unless-stopped
 mem_limit: 64M
 environment:
 GOMEMLIMIT: 110MiB
 privileged: true
 command:
 - --path.rootfs=/host
 - --web.listen-address=:9100
 network_mode: host
 pid: host
 volumes:
 - "/:/host:ro,rslave"

 podmanexporter:
 container_name: podmanexporter
 image: quay.io/navidys/prometheus-podman-exporter:v1.21.0
 mem_limit: 64M
 user: 0:0
 privileged: true
 environment:
 GOMEMLIMIT: 110MiB
 CONTAINER_HOST: unix:///run/podman/podman.sock
 restart: unless-stopped
 volumes:
 - /run/podman/podman.sock:/run/podman/podman.sock
 command:
 - --web.listen-address=:9882

 cgroupexporter:
 container_name: cgroupexporter
 image: ghcr.io/arianvp/cgroup-exporter:0.3.3-amd64
 mem_limit: 64M
 environment:
 GOMEMLIMIT: 110MiB
 restart: unless-stopped
 volumes:
 - /sys/fs/cgroup:/sys/fs/cgroup:ro
 command:
 - -listen-address=:3232

 pcm:
 container_name: pcm
 image: opcm/pcm
 mem_limit: 96M
 restart: unless-stopped
 privileged: true

Service CT

I keep public-facing services in a separate CT and forward only selected ports from the host.

Create an unprivileged Alpine CT on vmbr0, then:

1
2
3


apk add openssh openssh-server
rc-update add sshd default
rc-service sshd start

Add this to /etc/ssh/sshd_config:

1
2
3


PermitRootLogin yes
ClientAliveInterval 60
ClientAliveCountMax 3

Install Podman inside the CT:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


cat >/usr/local/bin/enablecgroup2nesting <<'EOF'
#!/bin/sh
if [ -f /sys/fs/cgroup/cgroup.controllers ]; then
 echo "Enabling cgroup v2 nesting"
 mkdir -p /sys/fs/cgroup/init
 xargs -rn1 < /sys/fs/cgroup/cgroup.procs > /sys/fs/cgroup/init/cgroup.procs || :
 sed -e 's/ / +/g' -e 's/^/+/' < /sys/fs/cgroup/cgroup.controllers \
 > /sys/fs/cgroup/cgroup.subtree_control
fi
EOF
chmod +x /usr/local/bin/enablecgroup2nesting

cat >/etc/init.d/cgroup2nesting <<'EOF'
#!/sbin/openrc-run

description="Enable nesting of cgroup2."

depend()
{
 keyword -docker -podman -prefix -systemd-nspawn -vserver -wsl
 after sysfs
}

start()
{
 ebegin "Enabling cgroup v2 nesting"
 /usr/local/bin/enablecgroup2nesting
 eend $?
}
EOF
chmod +x /etc/init.d/cgroup2nesting
rc-update add cgroup2nesting default
rc-service cgroup2nesting start

apk add podman iptables
sed -i 's/^#log_size_max =.*/log_size_max = 1048576/' /etc/containers/containers.conf

rc-update add podman
rc-service podman start

apk add docker-cli docker-cli-compose
docker context create podman --docker "host=unix:///run/podman/podman.sock"
docker context use podman

Also check out the Pi-Hole setup section for Alpine log cleanup and other small CT optimizations. You must do this because Alpine does not have log rotation by default. You disk will fill up if you do not manage logs.

Then run Caddy or Traefik in that CT for reverse proxy and TLS termination. Forward 80 and 443 from the host to the service CT with iptfwd or iptables.

If you use a privileged CT with Podman over ZFS, you may need an overlay mount helper:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


cat >/usr/local/bin/zfsoverlaymount <<'EOF'
#!/bin/sh
/bin/mount -t overlay overlay "$@"
EOF
chmod +x /usr/local/bin/zfsoverlaymount

cat >/etc/containers/storage.conf <<'EOF'
[storage.options]
mount_program = "/usr/local/bin/zfsoverlaymount"
EOF

I prefer unprivileged CTs unless there is a specific reason not to use them.

SMTP Notifications

By default, Proxmox may send mail directly and anonymously to the email address you entered during installation. Some providers reject that mail.

Configure a real SMTP target instead:

1

Datacenter -> Notifications -> Notification Targets -> Add -> SMTP

Final State

The host ended up with:

Proxmox VE 9.2.
Mirrored ZFS root pool on 128 GiB from each NVMe SSD.
Striped ZFS scratch pool on the remaining SSD space.
4K LBA on both NVMe drives.
Private-only vmbr0 for VMs and CTs.
IPv4 NAT and IPv6 NAT66 from vmbr0 to eno1.
80/443 forwarded to a service CT.
Pi-hole handling private DHCP and DNS.
Podman-based host monitoring.

For 19.90 USD/month, this is a surprisingly useful little virtualization box.

(Chinese Only) 华为 AirEngine 5773-21 踩坑记录

Sat, 13 Dec 2025 16:00:00 +0800

背景

最近老家重建，家里需要重新布置无线网络。考虑到自建别墅面积比较大，单纯使用单个无线路由器覆盖不够，因此多个 AP 肯定是需要的。家里在装修的时候就每个房间吊顶中有留超六类线缆，这种情况对 AP 来说是非常友好的，因此决定使用企业级无线接入点（AP）来组建无线网络。由于华为系列的 AP 在国内使用量较大，闲鱼上非常好捡公司下架下来的二手设备，因此考虑华为系列的 AP。这些企业级 AP 并不需要担心二手问题，他们的质量是消费级产品无法比拟的，稳定性也是非常强。

目前是 2025 年 12 月，虽然在国内 Wi-Fi 7 （802.11be）的 6 GHz 频段还没批准无法使用（损失不少性能），但是上一下 Wi-Fi 7 追一下新也没什么问题。华为 Wi-Fi 7 AP 是 AirEngine xx7x 产品，我闲鱼花了 500 多，激情下单了一台 AirEngine 5773-21 （有一个 2.5 Gbe PoE 口）。先买一台测测，为之后全屋部署积累经验，如果好用，再给每个房间买一台（后续更新）。

回来一查，这设备保修到 2028 年 3 月，现在才 2025 年 12 月，还有快 3 年的保修期，很新啊。

我目前的打算是不购买单独的 AC 控制器（AC 用于统一管理其他 AP ），因为我没有几十上百个 AP 需要管理。对于只需管理十几台 AP 来说，AC 控制器的成本和复杂度都不划算。因此打算使用华为 AP 自带的 Leader AP 模式组网（按道理华为的 AirEngine 系列大部分都是支持的，但是对于我买的 AirEngine 5773-21 来说，这里是个大坑，后面会讲）。这种模式下，Leader AP 可以担任 AC 的角色，统一管理其他 FIT AP ，每个 Leader AP 具体能管理几台 FIT AP 可以上华为 Info-Finder 上查询（基本上都大于 16 台）。

（组网方案后续更新，包括多 AP 的组网方案， VLAN 配置 IOT 设备隔离等）

首次启动

插上网线（注意，如果使用 PoE 的话需要满足 802.3at ，否则可能降速，或者使用 DC 供电）

注意，开机过程中状态灯常亮，等到状态灯快速闪烁说明开机完成等待配置。

现在华为的 AP 默认会用 DHCP 获取 IP 地址，因此只需根据 AP 的 MAC 地址（在 AP 背后会写）在 DHCP 服务器上找到对应的 IP 地址，就可以知道 AP 目前的 IP 地址了

然后浏览器打开该 IP 即可登录 Web 管理界面（并不）。

踩坑

然后浏览器打开该 IP 即可登录 Web 管理界面（吗？？？？），我发现并不能。

按道理华为 Wi-Fi 6 系列（比如 AirEngine 5761 ）都是可以直接登录 Web 管理页面的，Wi-Fi 7 更新不应该不行，而且官方 AirEngine 5773-21 的彩页中也说可以用做 FAT AP / Leader AP ，怎么回事？

我后面用 SSH 登录设备（后面会说怎么登录），没有说当前是 FIT 还是 FAT 模式（老的 AP 会在 SSH 登录的时候就提示）。而且这个命令行看的我一脸懵，老的 Wi-Fi 6 AP 都是直接 system-view 就能配置，然后命令也是传统的那一套。这新的 AP 一上来就是个 MDCLI> 提示符（一看就是新搞的东西，里面的命令也完全变了，我根本没时间去学习这些新命令。（后面才知道，必须升级系统后，用 switch cli 从 MC-CLI 切换到传统 CLI 模式，才能用老的命令行方式配置，但是建议还是用新的 MD-CLI ，因为传统 CLI 里面基本没啥功能了）

看了一圈文档，说可以输入 edit-config 修改配置，然后发现提示没有权限。WTF？我管理员还能没权限？后面才知道，原来没有权限其实代表当前 AP 在 FIT 模式下（这报错给用户带来多少困扰），必须切换到 FAT 模式才有权限修改配置。那么问题来了，我都没权限做任何配置，我怎么切换到 FAT 模式（况且我也没找到切换 FAT 模式的命令）？

后面折腾一圈才发现（各种根据文档中的蛛丝马迹去猜），是我当前固件版本太老了（V600R023C10），华为并没有在早期固件中实现 FAT AP / Leader AP 功能（设备都发布了，功能还没做好是吧），必须升级到 V600R024C10 及以上版本才有该功能，写文章时最新版本是 V600R025C00 。OK，问题找到，接下来就是升级固件了。

但是华为是出了名的不给资料，比如我想下载新版本固件，就算我注册了账号、也绑定了 AP 的序列号、也给了华为公司名称 + 设备序列号也审核通过了 AirEngine 5773-21 的资料和软件下载权限，但是你还是下不到 AirEngine 5773-21 的固件，华为官网上根本没有提供任何一个版本固件的下载链接（截至 2025 年 12 月），你只能下到补丁（这还是从其他 AP 找的补丁，刚好能用到 AirEngine 5773-21 上）。

最后我只能在闲鱼花钱上找有权限的人代下的固件，版本号 V600R024C10 ，也就是第一版支持 FAT AP / Leader AP 功能的固件。本来我想下载当前最新的 V600R025C00 ，但是我没找到人能下载这个版本，遂放弃， V600R024C10 也不是不能用。

首次设备设置

可以按住设备旁的 Default 按钮数十秒来重置设备。观察指示灯，变成常亮表示成功。重置成功后会重启，开机过程中状态灯常亮，等到状态灯快速闪烁说明开机完成等待配置。

你需要 SSH 上去 ssh admin@<AP_IP_ADDRESS> ，默认密码是 admin@huawei.com 。首次登录会提示你修改密码，按照提示修改后会登出，下次使用新密码再登录即可。

查看当前版本

1
2
3


[admin@HUAWEI]
MDCLI> display system/system-info/software-name
"AirEngineX773_V600R023C10SPC200.cc" # 当前版本

我目前的版本是 V600R023C10SPC200 ，也就是 V600R023C10 （后面的 SPC200 是补丁），需要升级到 V600R024C10 及以上版本才有 FAT AP / Leader AP 功能。

固件升级

首先需要找台机器开启 FTP 服务端，设置好用户名密码，打开读写权限（macOS App Store 中有个 QuickFTP 还是很好用的），到时候 AP 需要通过 FTP 上传和下载固件。

然后你需要有一个 V600R024C10 及以上版本的固件文件，放到 FTP 的根目录下面。

需要使用的固件名称通常长这样，两者都行：

AirEngineX773_V600R024C10.cc （不带补丁的基础固件）
AirEngineX773_V600R024C10SPC100.cc （带补丁的完整固件）

其中：

AirEngineX773：设备型号，比如 AirEngine 5773
V600R024C10：版本号，R 或者 C 之后的数字越大，版本越新。
SPC100：补丁号，可选。注意，不要下成 SPH 的热补丁了，补丁通常长这样 AirEngineX773_V600R024C10SPH150.pat（有 SPH 字样，且结尾是 .pat ），需要完整的固件（.cc 结尾）。

备份当前固件

SSH 登录设备操作

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


[admin@HUAWEI]
MDCLI> ftpc-transfer-file

[(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> command-type put

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> local-file-name AirEngineX773_V600R023C10SPC200.cc # 文件名与当前 AP 版本一致，查询：display system/system-info/software-name

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> remote-file-name AirEngineX773_V600R023C10SPC200.cc # 与上面保持一致即可

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> server-ipv4-address 192.168.213.54 # FTP 服务器 IP 地址

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> server-port 2121 # FTP 服务器端口

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> user-name xxxx # FTP 用户名

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> password # FTP 密码
Enter password:
Confirm password:

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> emit # 提交上传任务
{
 "huawei-ftpc:transfer-id": 3 # 可以用这个 ID 查看任务状态
}

[admin@HUAWEI]
MDCLI> display ftpc/transfer-tasks # 检查是否备份成功
{
 "transfer-task": [
 {
 "transfer-id": 3,
 "command-type": "put",
 "server-address": "192.168.213.54",
 "server-port": 2121,
 "local-file-name": "AirEngineX773_V600R023C10SPC200.cc",
 "remote-file-name": "AirEngineX773_V600R023C10SPC200.cc",
 "status": "succeeded", # 表示成功
 "percentage": 100
 }
 ]
}

你应该可以在 FTP 服务器上看到备份下来的固件文件了。

下载新固件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49


[admin@HUAWEI]
MDCLI> download-upgrade-package

[(x)admin@HUAWEI]/download-upgrade-package
MDCLI> base-software-directory AirEngineX773_V600R024C10SPC100.cc # 新固件文件名，注意要和 FTP 服务器上一致

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> server-ip 192.168.213.54 # FTP 服务器 IP 地址

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> server-port 2121 # FTP 服务器端口

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> transfer-protocol ftp # 传输协议

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> user-name xxxx # FTP 用户名

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> password # FTP 密码
Enter password:
Confirm password:

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> emit # 提交下载任务

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> display software/download-result/ # 查看下载状态
{
 "file-name": "AirEngineX773_V600R024C10SPC100.cc",
 "status": "succeeded", # 表示下载成功
 "percentage": 100
}

[admin@HUAWEI]
MDCLI> display file-operation/ # 查看文件列表，确认新固件在设备上
{
 "dir": [
...
 {
 "file-name": "AirEngineX773_V600R024C10SPC100.cc", # 新固件文件
 "dir-name": "backup:/",
 "attribute": "-rw-",
 "modify-time": "2024-11-12T12:05:04Z",
 "size": 52394852
 }
...
 ]
}

设置新固件为启动固件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


[admin@HUAWEI]
MDCLI> startup-by-mode name AirEngineX773_V600R024C10SPC100.cc # 设置启动固件，注意文件名要和上面一致

[admin@HUAWEI]
MDCLI> display cfg/startup-infos
{
 "startup-info": [
 {
 "position": "0",
 "configed-system-software": "AirEngineX773_V600R023C10SPC200.cc",
 "current-system-software": "AirEngineX773_V600R023C10SPC200.cc",
 "next-system-software": "AirEngineX773_V600R024C10SPC100.cc", # 确认新固件已设置为下次启动固件
 "current-cfg-file": "",
 "next-cfg-file": "",
 "current-patch-file": "NULL",
 "next-patch-file": "NULL"
 }
 ]
}

注意，要是你之前安装了 patch ，可能会导致新系统安装不上，例如

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


[admin@HUAWEI]
MDCLI> display cfg/startup-infos
{
 "startup-info": [
 {
 "position": "0",
 "configed-system-software": "AirEngineX773_V600R023C10SPC200.cc", # 是的，华为拼错单词了
 "current-system-software": "AirEngineX773_V600R023C10SPC200.cc",
 "next-system-software": "AirEngineX773_V600R023C10SPC200.cc", # 发现还是老的版本
 "current-cfg-file": "",
 "next-cfg-file": "",
 "current-patch-file": "AirEngineX773_V600R023SPH151.pat", # 已安装 patch
 "next-patch-file": "AirEngineX773_V600R023SPH151.pat"
 }
 ]
}

这时候你需要把 patch 给清掉：

1
2
3


[admin@HUAWEI]
MDCLI> delete-patch delete-type all
......

然后再设置启动固件即可。

重启设备

1
2
3
4


[admin@HUAWEI]
MDCLI> reboot
Warning: This operation will reboot the device.
Are you sure you want to continue? [Y(yes)/N(no)]:y

等设备起来，浏览器访问设备 IP 地址，应该就可以看到 Web 管理界面了。

选择右上角 FIT 按钮，改成 FAT 模式即可使用 Leader AP 功能，重启后即可变成 Leader AP ，可以管理其他 FIT AP 了。

安装补丁（Patch）

补丁是在基础版本上修复 bug 或者增加小功能的。补丁一般靠自己注册完设备就能在官网下到，下载补丁的时候，注意产品型号不需要选择 AirEngine 5773-21 ，因为太新了，华为官网上甚至没有这个型号。直接选择全部，只要版本对的上即可，比如我当前基础版本是 V600R024C10 ，那么补丁就需要选择 V600R024C10SPH181 这种补丁，表示在 V600R024C10 基础版本上打的热补丁 SPH181 ，数字越大表示补丁越新。补丁文件名通常长这样 AirEngineX773_V600R024C10SPH181.pat ，注意是 .pat 结尾。

下载后，放到 FTP 服务器根目录下，然后 SSH 登录设备，执行以下命令安装补丁：

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101


[admin@HUAWEI]
MDCLI> ftpc-transfer-file

[(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> command-type get

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> local-file-name AirEngineX773_V600R024C10SPH181.pat # 补丁文件名，注意和 FTP 服务器上一致

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> remote-file-name AirEngineX773_V600R024C10SPH181.pat # 与上面保持一致即可

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> server-ipv4-address 192.168.213.54 # FTP 服务器 IP 地址

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> server-port 2121 # FTP 服务器端口

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> user-name xxx # FTP 用户名

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> password # FTP 密码
Enter password:
Confirm password:

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> emit # 提交下载任务
{
 "huawei-ftpc:transfer-id": 1
}

[admin@HUAWEI]
MDCLI> display ftpc/transfer-tasks/ # 查看任务状态
{
 "transfer-task": [
 {
 "transfer-id": 1,
 "command-type": "get",
 "server-address": "192.168.213.54",
 "server-port": 2121,
 "local-file-name": "AirEngineX773_V600R024C10SPH181.pat",
 "remote-file-name": "AirEngineX773_V600R024C10SPH181.pat",
 "status": "succeeded", # 表示成功
 "percentage": 100
 }
 ]
}

[admin@HUAWEI]
MDCLI> load-patch name AirEngineX773_V600R024C10SPH181.pat load-type run # 安装补丁，注意文件名与需要安装的补丁一致

[admin@HUAWEI]
MDCLI> display patch/operation-schedules # 查看安装进度
{
 "operation-schedule": [
 {
 "phase": "load-patch",
 "status": "successful",
 "schedule": 100 # 等待进度 100%
 },
 {
 "phase": "delete-patch",
 "status": "not-started",
 "schedule": 0
 },
 {
 "phase": "startup-next-patch",
 "status": "not-started",
 "schedule": 0
 },
 {
 "phase": "reset-startup-patch",
 "status": "not-started",
 "schedule": 0
 }
 ]
}

[admin@HUAWEI]
MDCLI> display patch/patch-infos
{
 "patch-info": [
 {
 "name": "AirEngineX773_V600R024C10SPH181.pat", # 验证补丁安装成功
 "version": "V600R024C10SPH181",
 "state": "running",
 "runtime": "2025-12-13T21:39:48+08:00",
 "path": "/",
 "operations": {
 "operation": [
 {
 "position": "0",
 "position-type": "MPU",
 "upgrade-mode": "reset-board" # 某些补丁需要重启设备才能生效
 }
 ]
 }
 }
 ]
}

你也可以在 Web 管理界面上查看补丁是否安装成功。

Bonus: 启用 IPv6

我发现 AP 下面的设备获取不到 IPv6 地址（我内网中 IPv6 配置都是正确的），基本上肯定是在 AP 导致的 IPv6 地址无法下发。

后面了解华为 AP 默认不开 IPv6 报文转发。原因是：华为认为 IPv4 是主流，在 IPv4 网络中，如果存在较多 IPv6 协议报文，会影响无线网络性能，也会损耗设备的 CPU 处理能力。因此在纯 IPv4 网络中，可以通过不处理 IPv6 无线报文来提高 IPv4 网络性能。

不过，我就是要用 IPv6 ，因此要去把 WLAN 处理 IPv6 报文的功能打开（ Web UI 没有，要命令行开），SSH 登录设备，执行以下命令：

官方文档里面其实有这个，但是我找半天没找到，在：“参考-MD-CLI配置参考-WLAN配置-WLAN用户管理” 里面。不是，你把 IPv6 相关配置放在“用户管理”里面？这跟用户管理有啥关系？

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


[admin@HUAWEI]
MDCLI> edit-config

[(gl)admin@HUAWEI]
MDCLI> wlan-sta-access

[(gl)admin@HUAWEI]/wlan-sta-access
MDCLI> sta-ipv6-switch true

[*(gl)admin@HUAWEI]/wlan-sta-access
MDCLI> commit

之后你的设备应该就能正确收发 IPv6 报文了。

结语

这告诉我们， AirEngine 5773-21 这种 Wi-Fi 7 新设备，还是太新了，所有的坑都得自己踩。与 Wi-Fi 6 的设备比如 AirEngine 5761 相比网上大把教程还是差太远了。不过也算是积累了一些经验，后续再买多台 AP 的时候就不会再踩这些坑了。

这次单台 AP 就放北京用了，年底回老家再配置多 AP 组网方案。

Set Up ZeroTier Moon on Non-Standard Ports

Thu, 27 Nov 2025 10:20:00 +0800

To install ZeroTier, one would typically allow port 9993/UDP on their firewall. However, in certain scenarios, you may need to run a ZeroTier moon on non-standard ports due to network restrictions (e.g. behind NAT) or conflicts with other services. This guide will walk you through the steps to set up a ZeroTier moon on non-standard ports.

Setup Relays (Moon Nodes) on Non-Standard Ports

Install ZeroTier like usual:

1
2
3


curl -s https://install.zerotier.com | sudo bash
# and join a network (if your moon also acts as a client)
zerotier-cli join <network_id>

Setup moon:

1

zerotier-idtool initmoon /var/lib/zerotier-one/identity.public >>/var/lib/zerotier-one/moon.json

Here is the important part: edit the moon.json file to specify the desired non-standard ports. Open the file with your preferred text editor:

1

vim /var/lib/zerotier-one/moon.json

Modify the stableEndpoints section to include your public IP address along with the desired non-standard port. If you are behind a NAT, use your router’s public IP address and forward the same port from your server to your router. The format should be IP_ADDRESS/PORT.

For example, if you want to use port 14999, change the line to:

1

"stableEndpoints": ["xxx.xxx.xxx.xxx/14999"]

Generate the moon configuration (this is what clients will use to connect to your moon):

1

zerotier-idtool genmoon /var/lib/zerotier-one/moon.json

You should have one file that looks like *.moon in your current dir. Move the generated moon file to the ZeroTier directory:

1
2


mkdir -p /var/lib/zerotier-one/moons.d/
mv *.moon /var/lib/zerotier-one/moons.d/

If your moon also acts as a client, change the client configuration to use the non-standard port. Edit /var/lib/zerotier-one/local.conf:

1
2
3
4
5


{
 "settings": {
 "primaryPort": 14999
 }
}

Restart the ZeroTier service to apply the changes:

1

systemctl restart zerotier-one

Setup Clients (Leaf Nodes)

On the client side, you have to change the default port as well. Yes, the client’s default port (9993) must match the moon’s port. Otherwise, they won’t be able to communicate in my tests.

After you installed ZeroTier and joined the network, edit /var/lib/zerotier-one/local.conf:

1
2
3
4
5


{
 "settings": {
 "primaryPort": 14999
 }
}

Copy /var/lib/zerotier-one/moons.d/*.moon from the moon server to the client machine’s /var/lib/zerotier-one/moons.d/ directory, so the client knows about the moon:

1

scp "user@moon-server:/var/lib/zerotier-one/moons.d/*.moon" /var/lib/zerotier-one/moons.d/

Restart the ZeroTier service on the client:

1

systemctl restart zerotier-one

Verification

You can verify that the moon is functioning correctly by checking zerotier-cli command on the client to see if it can connect to the moon.

1

zerotier-cli peers

You should see an entry for your moon with the correct non-standard port, similar to the example below (xxx.xxx.xxx.xxx/14999):

1
2
3
4
5
6
7
8
9


zerotier-cli peers
200 peers
<ztaddr> <ver> <role> <lat> <link> <lastTX> <lastRX> <path>
35c192ce9b 1.15.3 LEAF 287 DIRECT 11575 11575 2001:19f0:6001:2c59:beef:3d:6767:df71/21006
3cdfac522d 1.16.0 MOON 66 DIRECT 3660 3660 xxx.xxx.xxx.xxx/14999
778cde7190 - PLANET 287 DIRECT 44090 43802 2605:9880:400:c3:254:f2bc:a1f7:19/9993
cafe04eba9 - PLANET 287 DIRECT 44090 43802 84.17.53.155/9993
cafe80ed74 - PLANET 261 DIRECT 269315 43837 2a02:6ea0:c87f::1/9993
cafefd6717 - PLANET 246 DIRECT 299345 43845 2a02:6ea0:d368::9993/9993

If your <lat> is -1, it means the client cannot reach the moon. Double-check your configuration and ensure that the specified ports are open and correctly forwarded if behind a NAT.