Networking on Charlie Chiang's blog

(Chinese Only) 华为 AirEngine 5773-21 踩坑记录

Sat, 13 Dec 2025 16:00:00 +0800

背景

最近老家重建，家里需要重新布置无线网络。考虑到自建别墅面积比较大，单纯使用单个无线路由器覆盖不够，因此多个 AP 肯定是需要的。家里在装修的时候就每个房间吊顶中有留超六类线缆，这种情况对 AP 来说是非常友好的，因此决定使用企业级无线接入点（AP）来组建无线网络。由于华为系列的 AP 在国内使用量较大，闲鱼上非常好捡公司下架下来的二手设备，因此考虑华为系列的 AP。这些企业级 AP 并不需要担心二手问题，他们的质量是消费级产品无法比拟的，稳定性也是非常强。

目前是 2025 年 12 月，虽然在国内 Wi-Fi 7 （802.11be）的 6 GHz 频段还没批准无法使用（损失不少性能），但是上一下 Wi-Fi 7 追一下新也没什么问题。华为 Wi-Fi 7 AP 是 AirEngine xx7x 产品，我闲鱼花了 500 多，激情下单了一台 AirEngine 5773-21 （有一个 2.5 Gbe PoE 口）。先买一台测测，为之后全屋部署积累经验，如果好用，再给每个房间买一台（后续更新）。

回来一查，这设备保修到 2028 年 3 月，现在才 2025 年 12 月，还有快 3 年的保修期，很新啊。

我目前的打算是不购买单独的 AC 控制器（AC 用于统一管理其他 AP ），因为我没有几十上百个 AP 需要管理。对于只需管理十几台 AP 来说，AC 控制器的成本和复杂度都不划算。因此打算使用华为 AP 自带的 Leader AP 模式组网（按道理华为的 AirEngine 系列大部分都是支持的，但是对于我买的 AirEngine 5773-21 来说，这里是个大坑，后面会讲）。这种模式下，Leader AP 可以担任 AC 的角色，统一管理其他 FIT AP ，每个 Leader AP 具体能管理几台 FIT AP 可以上华为 Info-Finder 上查询（基本上都大于 16 台）。

（组网方案后续更新，包括多 AP 的组网方案， VLAN 配置 IOT 设备隔离等）

首次启动

插上网线（注意，如果使用 PoE 的话需要满足 802.3at ，否则可能降速，或者使用 DC 供电）

注意，开机过程中状态灯常亮，等到状态灯快速闪烁说明开机完成等待配置。

现在华为的 AP 默认会用 DHCP 获取 IP 地址，因此只需根据 AP 的 MAC 地址（在 AP 背后会写）在 DHCP 服务器上找到对应的 IP 地址，就可以知道 AP 目前的 IP 地址了

然后浏览器打开该 IP 即可登录 Web 管理界面（并不）。

踩坑

然后浏览器打开该 IP 即可登录 Web 管理界面（吗？？？？），我发现并不能。

按道理华为 Wi-Fi 6 系列（比如 AirEngine 5761 ）都是可以直接登录 Web 管理页面的，Wi-Fi 7 更新不应该不行，而且官方 AirEngine 5773-21 的彩页中也说可以用做 FAT AP / Leader AP ，怎么回事？

我后面用 SSH 登录设备（后面会说怎么登录），没有说当前是 FIT 还是 FAT 模式（老的 AP 会在 SSH 登录的时候就提示）。而且这个命令行看的我一脸懵，老的 Wi-Fi 6 AP 都是直接 system-view 就能配置，然后命令也是传统的那一套。这新的 AP 一上来就是个 MDCLI> 提示符（一看就是新搞的东西，里面的命令也完全变了，我根本没时间去学习这些新命令。（后面才知道，必须升级系统后，用 switch cli 从 MC-CLI 切换到传统 CLI 模式，才能用老的命令行方式配置，但是建议还是用新的 MD-CLI ，因为传统 CLI 里面基本没啥功能了）

看了一圈文档，说可以输入 edit-config 修改配置，然后发现提示没有权限。WTF？我管理员还能没权限？后面才知道，原来没有权限其实代表当前 AP 在 FIT 模式下（这报错给用户带来多少困扰），必须切换到 FAT 模式才有权限修改配置。那么问题来了，我都没权限做任何配置，我怎么切换到 FAT 模式（况且我也没找到切换 FAT 模式的命令）？

后面折腾一圈才发现（各种根据文档中的蛛丝马迹去猜），是我当前固件版本太老了（V600R023C10），华为并没有在早期固件中实现 FAT AP / Leader AP 功能（设备都发布了，功能还没做好是吧），必须升级到 V600R024C10 及以上版本才有该功能，写文章时最新版本是 V600R025C00 。OK，问题找到，接下来就是升级固件了。

但是华为是出了名的不给资料，比如我想下载新版本固件，就算我注册了账号、也绑定了 AP 的序列号、也给了华为公司名称 + 设备序列号也审核通过了 AirEngine 5773-21 的资料和软件下载权限，但是你还是下不到 AirEngine 5773-21 的固件，华为官网上根本没有提供任何一个版本固件的下载链接（截至 2025 年 12 月），你只能下到补丁（这还是从其他 AP 找的补丁，刚好能用到 AirEngine 5773-21 上）。

最后我只能在闲鱼花钱上找有权限的人代下的固件，版本号 V600R024C10 ，也就是第一版支持 FAT AP / Leader AP 功能的固件。本来我想下载当前最新的 V600R025C00 ，但是我没找到人能下载这个版本，遂放弃， V600R024C10 也不是不能用。

首次设备设置

可以按住设备旁的 Default 按钮数十秒来重置设备。观察指示灯，变成常亮表示成功。重置成功后会重启，开机过程中状态灯常亮，等到状态灯快速闪烁说明开机完成等待配置。

你需要 SSH 上去 ssh admin@<AP_IP_ADDRESS> ，默认密码是 admin@huawei.com 。首次登录会提示你修改密码，按照提示修改后会登出，下次使用新密码再登录即可。

查看当前版本

1
2
3


[admin@HUAWEI]
MDCLI> display system/system-info/software-name
"AirEngineX773_V600R023C10SPC200.cc" # 当前版本

我目前的版本是 V600R023C10SPC200 ，也就是 V600R023C10 （后面的 SPC200 是补丁），需要升级到 V600R024C10 及以上版本才有 FAT AP / Leader AP 功能。

固件升级

首先需要找台机器开启 FTP 服务端，设置好用户名密码，打开读写权限（macOS App Store 中有个 QuickFTP 还是很好用的），到时候 AP 需要通过 FTP 上传和下载固件。

然后你需要有一个 V600R024C10 及以上版本的固件文件，放到 FTP 的根目录下面。

需要使用的固件名称通常长这样，两者都行：

AirEngineX773_V600R024C10.cc （不带补丁的基础固件）
AirEngineX773_V600R024C10SPC100.cc （带补丁的完整固件）

其中：

AirEngineX773：设备型号，比如 AirEngine 5773
V600R024C10：版本号，R 或者 C 之后的数字越大，版本越新。
SPC100：补丁号，可选。注意，不要下成 SPH 的热补丁了，补丁通常长这样 AirEngineX773_V600R024C10SPH150.pat（有 SPH 字样，且结尾是 .pat ），需要完整的固件（.cc 结尾）。

备份当前固件

SSH 登录设备操作

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


[admin@HUAWEI]
MDCLI> ftpc-transfer-file

[(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> command-type put

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> local-file-name AirEngineX773_V600R023C10SPC200.cc # 文件名与当前 AP 版本一致，查询：display system/system-info/software-name

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> remote-file-name AirEngineX773_V600R023C10SPC200.cc # 与上面保持一致即可

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> server-ipv4-address 192.168.213.54 # FTP 服务器 IP 地址

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> server-port 2121 # FTP 服务器端口

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> user-name xxxx # FTP 用户名

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> password # FTP 密码
Enter password:
Confirm password:

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> emit # 提交上传任务
{
 "huawei-ftpc:transfer-id": 3 # 可以用这个 ID 查看任务状态
}

[admin@HUAWEI]
MDCLI> display ftpc/transfer-tasks # 检查是否备份成功
{
 "transfer-task": [
 {
 "transfer-id": 3,
 "command-type": "put",
 "server-address": "192.168.213.54",
 "server-port": 2121,
 "local-file-name": "AirEngineX773_V600R023C10SPC200.cc",
 "remote-file-name": "AirEngineX773_V600R023C10SPC200.cc",
 "status": "succeeded", # 表示成功
 "percentage": 100
 }
 ]
}

你应该可以在 FTP 服务器上看到备份下来的固件文件了。

下载新固件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49


[admin@HUAWEI]
MDCLI> download-upgrade-package

[(x)admin@HUAWEI]/download-upgrade-package
MDCLI> base-software-directory AirEngineX773_V600R024C10SPC100.cc # 新固件文件名，注意要和 FTP 服务器上一致

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> server-ip 192.168.213.54 # FTP 服务器 IP 地址

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> server-port 2121 # FTP 服务器端口

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> transfer-protocol ftp # 传输协议

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> user-name xxxx # FTP 用户名

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> password # FTP 密码
Enter password:
Confirm password:

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> emit # 提交下载任务

[*(x)admin@HUAWEI]/download-upgrade-package
MDCLI> display software/download-result/ # 查看下载状态
{
 "file-name": "AirEngineX773_V600R024C10SPC100.cc",
 "status": "succeeded", # 表示下载成功
 "percentage": 100
}

[admin@HUAWEI]
MDCLI> display file-operation/ # 查看文件列表，确认新固件在设备上
{
 "dir": [
...
 {
 "file-name": "AirEngineX773_V600R024C10SPC100.cc", # 新固件文件
 "dir-name": "backup:/",
 "attribute": "-rw-",
 "modify-time": "2024-11-12T12:05:04Z",
 "size": 52394852
 }
...
 ]
}

设置新固件为启动固件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


[admin@HUAWEI]
MDCLI> startup-by-mode name AirEngineX773_V600R024C10SPC100.cc # 设置启动固件，注意文件名要和上面一致

[admin@HUAWEI]
MDCLI> display cfg/startup-infos
{
 "startup-info": [
 {
 "position": "0",
 "configed-system-software": "AirEngineX773_V600R023C10SPC200.cc",
 "current-system-software": "AirEngineX773_V600R023C10SPC200.cc",
 "next-system-software": "AirEngineX773_V600R024C10SPC100.cc", # 确认新固件已设置为下次启动固件
 "current-cfg-file": "",
 "next-cfg-file": "",
 "current-patch-file": "NULL",
 "next-patch-file": "NULL"
 }
 ]
}

注意，要是你之前安装了 patch ，可能会导致新系统安装不上，例如

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


[admin@HUAWEI]
MDCLI> display cfg/startup-infos
{
 "startup-info": [
 {
 "position": "0",
 "configed-system-software": "AirEngineX773_V600R023C10SPC200.cc", # 是的，华为拼错单词了
 "current-system-software": "AirEngineX773_V600R023C10SPC200.cc",
 "next-system-software": "AirEngineX773_V600R023C10SPC200.cc", # 发现还是老的版本
 "current-cfg-file": "",
 "next-cfg-file": "",
 "current-patch-file": "AirEngineX773_V600R023SPH151.pat", # 已安装 patch
 "next-patch-file": "AirEngineX773_V600R023SPH151.pat"
 }
 ]
}

这时候你需要把 patch 给清掉：

1
2
3


[admin@HUAWEI]
MDCLI> delete-patch delete-type all
......

然后再设置启动固件即可。

重启设备

1
2
3
4


[admin@HUAWEI]
MDCLI> reboot
Warning: This operation will reboot the device.
Are you sure you want to continue? [Y(yes)/N(no)]:y

等设备起来，浏览器访问设备 IP 地址，应该就可以看到 Web 管理界面了。

选择右上角 FIT 按钮，改成 FAT 模式即可使用 Leader AP 功能，重启后即可变成 Leader AP ，可以管理其他 FIT AP 了。

安装补丁（Patch）

补丁是在基础版本上修复 bug 或者增加小功能的。补丁一般靠自己注册完设备就能在官网下到，下载补丁的时候，注意产品型号不需要选择 AirEngine 5773-21 ，因为太新了，华为官网上甚至没有这个型号。直接选择全部，只要版本对的上即可，比如我当前基础版本是 V600R024C10 ，那么补丁就需要选择 V600R024C10SPH181 这种补丁，表示在 V600R024C10 基础版本上打的热补丁 SPH181 ，数字越大表示补丁越新。补丁文件名通常长这样 AirEngineX773_V600R024C10SPH181.pat ，注意是 .pat 结尾。

下载后，放到 FTP 服务器根目录下，然后 SSH 登录设备，执行以下命令安装补丁：

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101


[admin@HUAWEI]
MDCLI> ftpc-transfer-file

[(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> command-type get

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> local-file-name AirEngineX773_V600R024C10SPH181.pat # 补丁文件名，注意和 FTP 服务器上一致

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> remote-file-name AirEngineX773_V600R024C10SPH181.pat # 与上面保持一致即可

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> server-ipv4-address 192.168.213.54 # FTP 服务器 IP 地址

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> server-port 2121 # FTP 服务器端口

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> user-name xxx # FTP 用户名

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> password # FTP 密码
Enter password:
Confirm password:

[*(x)admin@HUAWEI]/ftpc-transfer-file
MDCLI> emit # 提交下载任务
{
 "huawei-ftpc:transfer-id": 1
}

[admin@HUAWEI]
MDCLI> display ftpc/transfer-tasks/ # 查看任务状态
{
 "transfer-task": [
 {
 "transfer-id": 1,
 "command-type": "get",
 "server-address": "192.168.213.54",
 "server-port": 2121,
 "local-file-name": "AirEngineX773_V600R024C10SPH181.pat",
 "remote-file-name": "AirEngineX773_V600R024C10SPH181.pat",
 "status": "succeeded", # 表示成功
 "percentage": 100
 }
 ]
}

[admin@HUAWEI]
MDCLI> load-patch name AirEngineX773_V600R024C10SPH181.pat load-type run # 安装补丁，注意文件名与需要安装的补丁一致

[admin@HUAWEI]
MDCLI> display patch/operation-schedules # 查看安装进度
{
 "operation-schedule": [
 {
 "phase": "load-patch",
 "status": "successful",
 "schedule": 100 # 等待进度 100%
 },
 {
 "phase": "delete-patch",
 "status": "not-started",
 "schedule": 0
 },
 {
 "phase": "startup-next-patch",
 "status": "not-started",
 "schedule": 0
 },
 {
 "phase": "reset-startup-patch",
 "status": "not-started",
 "schedule": 0
 }
 ]
}

[admin@HUAWEI]
MDCLI> display patch/patch-infos
{
 "patch-info": [
 {
 "name": "AirEngineX773_V600R024C10SPH181.pat", # 验证补丁安装成功
 "version": "V600R024C10SPH181",
 "state": "running",
 "runtime": "2025-12-13T21:39:48+08:00",
 "path": "/",
 "operations": {
 "operation": [
 {
 "position": "0",
 "position-type": "MPU",
 "upgrade-mode": "reset-board" # 某些补丁需要重启设备才能生效
 }
 ]
 }
 }
 ]
}

你也可以在 Web 管理界面上查看补丁是否安装成功。

Bonus: 启用 IPv6

我发现 AP 下面的设备获取不到 IPv6 地址（我内网中 IPv6 配置都是正确的），基本上肯定是在 AP 导致的 IPv6 地址无法下发。

后面了解华为 AP 默认不开 IPv6 报文转发。原因是：华为认为 IPv4 是主流，在 IPv4 网络中，如果存在较多 IPv6 协议报文，会影响无线网络性能，也会损耗设备的 CPU 处理能力。因此在纯 IPv4 网络中，可以通过不处理 IPv6 无线报文来提高 IPv4 网络性能。

不过，我就是要用 IPv6 ，因此要去把 WLAN 处理 IPv6 报文的功能打开（ Web UI 没有，要命令行开），SSH 登录设备，执行以下命令：

官方文档里面其实有这个，但是我找半天没找到，在：“参考-MD-CLI配置参考-WLAN配置-WLAN用户管理” 里面。不是，你把 IPv6 相关配置放在“用户管理”里面？这跟用户管理有啥关系？

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


[admin@HUAWEI]
MDCLI> edit-config

[(gl)admin@HUAWEI]
MDCLI> wlan-sta-access

[(gl)admin@HUAWEI]/wlan-sta-access
MDCLI> sta-ipv6-switch true

[*(gl)admin@HUAWEI]/wlan-sta-access
MDCLI> commit

之后你的设备应该就能正确收发 IPv6 报文了。

结语

这告诉我们， AirEngine 5773-21 这种 Wi-Fi 7 新设备，还是太新了，所有的坑都得自己踩。与 Wi-Fi 6 的设备比如 AirEngine 5761 相比网上大把教程还是差太远了。不过也算是积累了一些经验，后续再买多台 AP 的时候就不会再踩这些坑了。

这次单台 AP 就放北京用了，年底回老家再配置多 AP 组网方案。

Set Up ZeroTier Moon on Non-Standard Ports

Thu, 27 Nov 2025 10:20:00 +0800

To install ZeroTier, one would typically allow port 9993/UDP on their firewall. However, in certain scenarios, you may need to run a ZeroTier moon on non-standard ports due to network restrictions (e.g. behind NAT) or conflicts with other services. This guide will walk you through the steps to set up a ZeroTier moon on non-standard ports.

Setup Relays (Moon Nodes) on Non-Standard Ports

Install ZeroTier like usual:

1
2
3


curl -s https://install.zerotier.com | sudo bash
# and join a network (if your moon also acts as a client)
zerotier-cli join <network_id>

Setup moon:

1

zerotier-idtool initmoon /var/lib/zerotier-one/identity.public >>/var/lib/zerotier-one/moon.json

Here is the important part: edit the moon.json file to specify the desired non-standard ports. Open the file with your preferred text editor:

1

vim /var/lib/zerotier-one/moon.json

Modify the stableEndpoints section to include your public IP address along with the desired non-standard port. If you are behind a NAT, use your router’s public IP address and forward the same port from your server to your router. The format should be IP_ADDRESS/PORT.

For example, if you want to use port 14999, change the line to:

1

"stableEndpoints": ["xxx.xxx.xxx.xxx/14999"]

Generate the moon configuration (this is what clients will use to connect to your moon):

1

zerotier-idtool genmoon /var/lib/zerotier-one/moon.json

You should have one file that looks like *.moon in your current dir. Move the generated moon file to the ZeroTier directory:

1
2


mkdir -p /var/lib/zerotier-one/moons.d/
mv *.moon /var/lib/zerotier-one/moons.d/

If your moon also acts as a client, change the client configuration to use the non-standard port. Edit /var/lib/zerotier-one/local.conf:

1
2
3
4
5


{
 "settings": {
 "primaryPort": 14999
 }
}

Restart the ZeroTier service to apply the changes:

1

systemctl restart zerotier-one

Setup Clients (Leaf Nodes)

On the client side, you have to change the default port as well. Yes, the client’s default port (9993) must match the moon’s port. Otherwise, they won’t be able to communicate in my tests.

After you installed ZeroTier and joined the network, edit /var/lib/zerotier-one/local.conf:

1
2
3
4
5


{
 "settings": {
 "primaryPort": 14999
 }
}

Copy /var/lib/zerotier-one/moons.d/*.moon from the moon server to the client machine’s /var/lib/zerotier-one/moons.d/ directory, so the client knows about the moon:

1

scp "user@moon-server:/var/lib/zerotier-one/moons.d/*.moon" /var/lib/zerotier-one/moons.d/

Restart the ZeroTier service on the client:

1

systemctl restart zerotier-one

Verification

You can verify that the moon is functioning correctly by checking zerotier-cli command on the client to see if it can connect to the moon.

1

zerotier-cli peers

You should see an entry for your moon with the correct non-standard port, similar to the example below (xxx.xxx.xxx.xxx/14999):

1
2
3
4
5
6
7
8
9


zerotier-cli peers
200 peers
<ztaddr> <ver> <role> <lat> <link> <lastTX> <lastRX> <path>
35c192ce9b 1.15.3 LEAF 287 DIRECT 11575 11575 2001:19f0:6001:2c59:beef:3d:6767:df71/21006
3cdfac522d 1.16.0 MOON 66 DIRECT 3660 3660 xxx.xxx.xxx.xxx/14999
778cde7190 - PLANET 287 DIRECT 44090 43802 2605:9880:400:c3:254:f2bc:a1f7:19/9993
cafe04eba9 - PLANET 287 DIRECT 44090 43802 84.17.53.155/9993
cafe80ed74 - PLANET 261 DIRECT 269315 43837 2a02:6ea0:c87f::1/9993
cafefd6717 - PLANET 246 DIRECT 299345 43845 2a02:6ea0:d368::9993/9993

If your <lat> is -1, it means the client cannot reach the moon. Double-check your configuration and ensure that the specified ports are open and correctly forwarded if behind a NAT.

Rename RDMA NIC Interface Names

Fri, 09 Aug 2024 14:28:00 +0800

Background

The RDMA NIC names (InfiniBand HCAs in this case) of our nodes are inconsistent. We want to rename them to a consistent naming scheme.

One node:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81


root@tj01-h20-node139:/# ibstatus
Infiniband device 'ibp171s0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x70
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'ibp187s0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x6f
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'ibp203s0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x29
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'ibp219s0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x6a
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'ibp41s0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x6c
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'ibp59s0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x2b
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'ibp75s0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x80
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'ibp93s0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x6e
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'rocep22s0f0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x0
 sm lid: 0x0
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 25 Gb/sec (1X EDR)
 link_layer: Ethernet

Another node:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81


root@tj01-h20-node140:/# ibstatus
Infiniband device 'mlx5_2' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x68
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'mlx5_3' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x71
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'mlx5_4' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x27
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'mlx5_5' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x69
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'mlx5_6' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x6d
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'mlx5_7' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x6b
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'mlx5_8' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x2a
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'mlx5_9' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x2c
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Infiniband device 'mlx5_bond_0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x0
 sm lid: 0x0
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 25 Gb/sec (1X EDR)
 link_layer: Ethernet

Note that one node has the names ibp* and the other has mlx5_*. We want to make node 1 follow a same naming scheme as the other nodes.

To be fair ibp* is actually consistent (named using PCI location). mlx5_* is not consistent, which depends on the card initialization order and can change after a reboot.

Anyway, since all other nodes are using mlx5_* naming scheme, we want to rename all of them to mlx5_* for consistency with the other nodes.

Solution

1

cp cp /lib/udev/rules.d/60-rdma-persistent-naming.rules /etc/udev/rules.d/

This is all you need. Now reboot your server.

If you are interested in the details, read on.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


# SPDX-License-Identifier: (GPL-2.0 OR Linux-OpenIB)
# Copyright (c) 2019, Mellanox Technologies. All rights reserved. See COPYING file
#
# Rename modes:
# NAME_FALLBACK - Try to name devices in the following order:
# by-pci -> by-guid -> kernel
# NAME_KERNEL - leave name as kernel provided
# NAME_PCI - based on PCI/slot/function location
# NAME_GUID - based on system image GUID
# NAME_FIXED - rename the device to the fixed named in the next argument
#
# The stable names are combination of device type technology and rename mode.
# Infiniband - ib*
# RoCE - roce*
# iWARP - iw*
# OPA - opa*
# Default (unknown protocol) - rdma*
#
# Example:
# * NAME_PCI
# pci = 0000:00:0c.4
# Device type = IB
# mlx5_0 -> ibp0s12f4
# * NAME_GUID
# GUID = 5254:00c0:fe12:3455
# Device type = RoCE
# mlx5_0 -> rocex525400c0fe123455
#
ACTION=="add", SUBSYSTEM=="infiniband", PROGRAM="rdma_rename %k NAME_KERNEL"

# Example:
# * NAME_FIXED
# fixed name for specific board_id
#
#ACTION=="add", ATTR{board_id}=="MSF0010110035", SUBSYSTEM=="infiniband", PROGRAM="rdma_rename %k NAME_FIXED myib"

Look at the line contains PROGRAM="rdma_rename %k NAME_KERNEL". So if you want to use mlx5_* (called kernel names), you can use NAME_KERNEL as the rename mode (default). If you want to use ibp* (called PCI names), you can use NAME_PCI as the rename mode or use NAME_FALLBACK, which first tries NAME_PCI.

NFS over RDMA Setup & Benchmark

Fri, 19 Jul 2024 22:36:00 +0800

So I decided to test NFS over RDMA somehow.

Hostname	`ft2000` (Server)	`t3640` (Client)
CPU	Phytium FT-2000+/64	Intel Core i9-10900K
RAM	Quad Channel DDR4 128GB	Dual Channel DDR4 64GB
NIC	Mellanox ConnextX-4 Lx 25GbE	Mellanox ConnextX-4 Lx 25GbE
OS	UOS Server 20 1070a	Debian 12

Two servers are connected using a 25GbE network. The goal is to set up NFS over RDMA and benchmark it to see the performance difference between NFS over TCP and NFS over RDMA.

RDMA Link Verification

Install RDMA/IB dependencies:

Dependencies should be the same on both devices (no server and client differences). The difference is only the package manager (and OS).

If you are using Mellanox OFED, you can skip this step. Be sure to install Mellanox OFED using --with-nfsrdma flag. Otherwise, you will not be able to use NFS over RDMA.

On UOS Server 20 1070a, which is Anolis8-based, which is again CentOS8-based:

1
2
3
4


root@ft2000:/# yum install -y libibverbs librdmacm opensm-libs rdma-core rdma-core-devel \
 librdmacm-utils opensm-static srp_daemon ucx-devel ucx-rdmacm infiniband-diags ibacm \
 opensm-devel ucx ucx-ib libibumad opensm mstflint ucx-cma openmpi rshim libibverbs-utils \
 perftest rdma-core rdma-core-devel infiniband-diags rshim

On Debian 12:

1
2
3
4


root@t3640:/# apt install -y infiniband-diags srptools perftest opensm-doc librdmacm-dev \
 rdmacm-utils librdmacm1 ibacm libibmad-dev libibmad5 libibumad-dev libibumad3 \
 ibverbs-utils libibverbs-dev libibverbs1 mstflint rdma-core opensm fio librbd1 \
 librados2 libibnetdisc5 ibverbs-providers

Make sure link is up on both servers using ip or ibstatus command:

Client:

Use ip (note state UP):

1
2


root@t3640:/# ip l show dev enp1s0f1np1
6: enp1s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

Or use ibstatis (note phys state: 5: LinkUp):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


root@t3640:/# ibstatus
Infiniband device 'mlx5_0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x0
 sm lid: 0x0
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 10 Gb/sec (1X QDR)
 link_layer: Ethernet

Infiniband device 'mlx5_1' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x0
 sm lid: 0x0
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 25 Gb/sec (1X EDR)
 link_layer: Ethernet

Server:

Use ip (note state UP):

1
2


root@ft2000:/# ip l show dev enp16s0f0np0
4: enp16s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

Or use ibstatis (note phys state: 5: LinkUp):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


root@ft2000:/# ibstatus
Infiniband device 'mlx5_0' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x0
 sm lid: 0x0
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 25 Gb/sec (1X EDR)
 link_layer: Ethernet

Infiniband device 'mlx5_1' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x0
 sm lid: 0x0
 state: 1: DOWN
 phys state: 3: Disabled
 rate: 40 Gb/sec (4X QDR)
 link_layer: Ethernet

Verify IP connectivity between the servers (ping the IP address of the other server):

1
2
3
4


root@t3640:/# ping 192.168.23.1
PING 192.168.23.1 (192.168.23.1) 56(84) bytes of data.
64 bytes from 192.168.23.1: icmp_seq=1 ttl=64 time=0.199 ms
64 bytes from 192.168.23.1: icmp_seq=2 ttl=64 time=0.206 ms

Make sure the the InfiniBand kernel modules are enabled.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# On both servers
root@ft2000:/# lsmod | grep ^ib
ib_srpt 262144 0
ib_isert 262144 0
ib_iser 262144 0
ib_umad 262144 0
ib_ipoib 327680 0
ib_cm 262144 3 rdma_cm,ib_ipoib,ib_srpt
ib_uverbs 327680 2 rdma_ucm,mlx5_ib
ib_core 458752 12 rdma_cm,ib_ipoib,rpcrdma,ib_srpt,iw_cm,ib_iser,ib_umad,ib_isert,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm

Make sure you have a lossless network:

In case the RDMA is running over Ethernet (RoCE) you need to make sure that the network is configured to be loss-less, which means that either flow control (FC) or priority flow control PFC is enabled on the adapter ports and the switch.

In case of lab environment or small setup, you can use Global Pause Flow Control to create loss-less environment. To check what is the global pause configuration use the following command (by default it is enabled normally).

1
2
3
4
5


root@t3640:/# ethtool -a enp1s0f0np0
Pause parameters for enp1s0f0np0:
Autonegotiate: off
RX: on
TX: on

In case it is disabled, run:

1

root@t3640:/# ethtool -A enp1s0f0np0 rx on tx on

Test RDMA speed. Refer to my previous blog InfiniBand Performance Test for details.

Run the following command on client and server, respectively:

1
2


root@t3640:/# ib_send_bw --report_gbit -F -d mlx5_1
# device name can be seen in `ibstatus` command

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


root@ft2000:/# ib_send_bw --report_gbit -a -F -d mlx5_0 192.168.23.2
---------------------------------------------------------------------------------------
 Send BW Test
 Dual-port : OFF Device : mlx5_0
 Number of qps : 1 Transport type : IB
 Connection type : RC Using SRQ : OFF
 PCIe relax order: ON
 ibv_wr* API : ON
 TX depth : 128
 CQ Moderation : 100
 Mtu : 1024[B]
 Link type : Ethernet
 GID index : 3
 Max inline data : 0[B]
 rdma_cm QPs : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x0087 PSN 0x176d37
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:23:01
 remote address: LID 0000 QPN 0x0188 PSN 0x789e84
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:23:02
---------------------------------------------------------------------------------------
 #bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
 65536 1000 23.14 23.13 0.044119
---------------------------------------------------------------------------------------

As you can see, the speed is around 23 Gb/sec. This is the maximum speed of the network (25G). If you see a lower speed, you need to check the network configuration and the switch configuration.

Configure NFS without RDMA

Just to make sure NFS is working.

Install NFS on server:

1
2
3


root@ft2000:/# yum -y install nfs-utils # On Debian: apt install nfs-kernel-server
root@ft2000:/# systemctl start nfs-server rpcbind
root@ft2000:/# systemctl enable nfs-server rpcbind

Use /tmp as a test dir and share it:

Since I want to test the network performance (TCP and RDMA), I don’t want to be bottlenecked by the disk speed. So I will use /tmp, which is backed by RAM (usually), as a reasonably fast directory to avoid bottlenecks.

Make sure it is tmpfs, otherwise IO will be limited by your disk speed.
1
2
root@ft2000:/# mount | grep /tmp
tmpfs on /tmp type tmpfs (rw,nosuid,nodev)

1

root@ft2000:/# echo '/tmp *(rw,async,insecure,no_root_squash)' >>/etc/exports

Make it available:

1
2


root@ft2000:/# exportfs -a
root@ft2000:/# systemctl restart nfs-server

Install NFS on client:

1

root@t3640:/# apt install nfs-common

Mount the NFS share temporarily:

1
2


root@t3640:/# mkdir -p /mnt/nfs
root@t3640:/# mount -t nfs 192.168.23.1:/tmp /mnt/nfs

Verify the mount:

Should see the files in /tmp on the server.

1

root@t3640:/# ls -l /mnt/nfs

Configure NFS with RDMA

Load RDMA transport module on the server:

1
2
3


root@ft2000:/# modprobe svcrdma
# To make it permanent: echo 'svcrdma' >>/etc/modules
root@ft2000:/# systemctl restart nfs-server

Instruct the server to listen on the RDMA transport port (20049 is the default port):

Note: if you see echo: write error: protocol not supported, it means that the NFSoRDMA is not supported in the Mellanox OFED. You need to use the inbox OS driver (refer to Install RDMA/IB dependencies section in this blog) or re-install Mellanox OFED with --with-nfsrdma flag. I was tripped by this issue.

1
2


root@ft2000:/# echo rdma 20049 > /proc/fs/nfsd/portlist
# To make it permanent: echo 'rdma=nfsrdma' >> /etc/nfs.conf

Load the RDMA transport module on the client

1
2
3


root@t3640:/# modprobe xprtrdma
# To make it permanent: echo 'xprtrdma' >>/etc/modules
root@t3640:/# systemctl restart nfs-utils.service

Mount the NFS share with RDMA:

Before running the mount command, unmount the NFS share if it is mounted (umount /mnt/nfs).

1
2


root@t3640:/# mount -t nfs -o rdma,port=20049 192.168.23.1:/tmp /mnt/nfs
# To make it permanent: echo '192.168.23.1:/tmp /mnt/nfs nfs defaults,proto=rdma,port=20049 0 0' >> /etc/fstab

Check the mount parameters:

If you see proto=rdma, it means the NFS is mounted using RDMA.

1
2
3


root@t3640:/# nfsstat -m
/mnt/nfs from 192.168.23.1:/tmp
 Flags: rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.23.2,local_lock=none,addr=192.168.23.1

Verify the mount:

Should see the files in /tmp on the server.

1

root@t3640:/# ls -l /mnt/nfs

Benchmark

Use fio to benchmark the NFS performance. Parameters such as bs, iodepth will be changed to different values later to evaluate the performance in different scenarios.

I will only test random read performance since sequential read performance is not a main concern even using TCP let alone RDMA.

The actual command will be:

1
2
3


root@t3640:/# fio --rw=randread --bs=4k --numjobs=1 --iodepth=1 --runtime=10 --time_based --loops=1 \
 --ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap \
 --exitall --name task1 --filename=/mnt/nfs/testfile --size=256M

The full log and related scripts can be found in the Appendix.

Here are the results we have all been waiting for!

For IO operations per second (IOPS), the average IOPS for NFS over RDMA is higher than NFS over TCP. The difference is more significant when the block size is larger and the most significant when iodepth=12. We can also see that no matter the block size and iodepth, the IOPS for NFS over TCP is capped at around 47 KIOPS, while the one of RDMA is significantly higher at over 180 KIOPS.

The reason why the difference is smaller with larger block sizes is that the network bandwidth is the bottleneck. For our 25GbE network, the maximum bandwidth is around 2800 MiB/s. We are hitting this limit with larger block sizes and this much IOPS. This is unrelated to the RDMA or TCP. If you have a faster network, you will see this difference for larger block sizes as well.

For 4K block size:

IO Depth	KIOPS (RDMA)	KIOPS (TCP)	Ratio
8	124.5	36.1	3.4
12	156.0	38.1	4.1
16	165.8	41.4	4.0
24	177.0	45.8	3.9
32	179.4	50.7	3.5

For bandwidth, the average bandwidth for NFS over RDMA is significantly higher (around 4x for small block sizes).

The reason why the difference is more significant when the block size is small (4-16) is that we are limited by the network bandwidth. Since we have a 25GbE network, the maximum bandwidth is around 2800 MiB/s so you can see that the bandwidth is capped at 2800 MiB/s for large block sizes and deep IO depths. If you have a faster network, you will see this 4x difference for larger block sizes as well.

Latency-wise, the average latency for NFS over RDMA is slightly lower than NFS over TCP. The difference is more significant for small block size and large iodepths.

Appendix

Benchmark script benchmark.sh: automatically run fio with different bs and iodepth values and saves logs (it saves stdout, not the fio-generated log, e.g. --bandwidth-log).
Plotting script plot.py: parses the fio logs (stdout) saved by benchmark.sh and plots the results, which is the three figures you see above in the Benchmark section.

Raw fio Logs (Click to Show)

Flash 10G CX4121A to 25G

Fri, 19 Jul 2024 16:37:00 +0800

10G ConnectX-4 Lx NICs can be flashed to 25G with the right firmware. This is useful if you have a 10G NIC and want to upgrade to 25G without buying a new NIC. This article will guide you through the process of flashing a 10G Mellanox ConnectX-4 Lx NIC to 25G.

Warning: Do not try this on MCX4121C. It will brick the NIC.

Download firmware from Firmware for ConnectX®-4 Lx EN. Choose MCX4121A-ACUT. I will assume the firmware file is named fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACU_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin.

If you are wondering what’s the difference between models, here’s a table for ConnectX-4 Lx cards (original document). You should choose 25G models to flash to.

Max Network Speed	Interface Type	Supported Ethernet Speeds (GbE)	Host Interface	Additional Features	OPN
1x 10GbE	SFP28	10, 1	PCIe 3.0 x8		MCX4111A-XCAT
2x 10GbE	SFP28	10, 1	PCIe 3.0 x8		MCX4121A-XCAT
2x 10GbE	SFP28	10, 1	PCIe 3.0 x8	Host Management, UEFI Enabled	MCX4121A-XCHT
1x 25GbE	SFP28	25, 10, 1	PCIe 3.0 x8		MCX4111A-ACAT
1x 25GbE	SFP28	25, 10, 1	PCIe 3.0 x8	UEFI Enabled	MCX4111A-ACUT
2x 25GbE	SFP28	25, 10, 1	PCIe 3.0 x8		MCX4121A-ACAT
2x 25GbE	SFP28	25, 10, 1	PCIe 3.0 x8	UEFI Enabled	MCX4121A-ACUT
1x 40GbE	QSFP28	40, 25, 10, 1	PCIe 3.0 x8		MCX4131A-BCAT
1x 50GbE	QSFP28	50, 40, 25, 10, 1	PCIe 3.0 x8		MCX4131A-GCAT

Download and install NVIDIA Firmware Tools (MFT).
mst start
mst ststus You will see you MST device, e.g. /dev/mst/mt4117_pciconf0. I will use this device path in this article. If your path differs from this, please change it accordingly.
Save GUID, MAC: flint -d /dev/mst/mt4117_pciconf0 query full > flint_query.txt
Save hardware info: flint -d /dev/mst/mt4117_pciconf0 hw query > flint_hwinfo.txt
Save current firmware: flint -d /dev/mst/mt4117_pciconf0 ri orig_firmware.bin
Save current firmware config: flint -d /dev/mst/mt4117_pciconf0 dc orig_firmware.ini
Save current PXE ROM (if exists): flint -d /dev/mst/mt4117_pciconf0 rrom orig_rom.bin
Save current PCI VPD: mlxburn -d /dev/mst/mt4117_pciconf0 -vpd > orig_vpd.txt
flint -i fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACU_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin verify
flint -i fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACU_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin -d /dev/mst/mt4117_pciconf0 -allow_psid_change burn

Reboot you machine. You should see your NIC is now have 25G speed.

Bonus:

To change MAC address: flint -d /dev/mst/mt4117_pciconf0 -mac 02c90abcdef0 sg
To change GUID: flint -d /dev/mst/mt4117_pciconf0 -guid 0002c9000abcdef0 sg

InfiniBand Performance Test

Fri, 19 Jul 2024 11:10:00 +0800

We have a 200Gbps InfiniBand network among about 100 nodes, connected using NVIDIA ConnectX-7 NICs (HDR200) to a NVIDIA MQM9700 switch. Each OSFP twin port (2xNDR, 2x400Gbps) on QM9700 is splitted into 4xNDR200 ports to connect to 4xNVIDIA ConnectX-7 NICs using 2xNDR to 4xNDR200 DAC/ACC (OSFP to 4x OSFP).

We want to test the performance of the network between 2 nodes just to make sure it is working. We expect to see a bandwidth of around 200Gbps.

Check the status of the Infiniband devices. Make sure at least one link is up (phys state: 5: LinkUp) on both nodes.

1
2
3
4
5
6
7
8
9


root@tj01-h20-node139:/# ibstatus
Infiniband device 'mlx5_7' port 1 status:
 default gid: fe80:0000:0000:0000:<redacted>
 base lid: 0x6c
 sm lid: 0x4
 state: 4: ACTIVE
 phys state: 5: LinkUp
 rate: 200 Gb/sec (2X NDR)
 link_layer: InfiniBand

Check if both nodes are connected to the Subnet Manager. You should see the name of both nodes appear on the list.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


root@tj01-h20-node139:/# ibnetdiscover
#
# Topology file: generated on Fri Jul 19 02:49:25 2024
#
# Initiated from node <redacted> port <redacted>

vendid=0x2c9
devid=0xd2f2
sysimgguid=0x9c05<redacted>
switchguid=0x9c05<redacted>(9c05<redacted>)
Switch 129 "S-9c05<redacted>" # "MF0;CNTSN-POD229-QM97-IB-02:MQM9700/U1" enhanced port 0 lid 1 lmc 0
[1] "H-b83f<redacted>"[1](b83f<redacted>) # "tj01-4090-node005 mlx5_2" lid 22 2xNDR
[2] "H-946d<redacted>"[1](946d<redacted>) # "tj01-4090-node004 mlx5_2" lid 27 2xNDR
[3] "H-a088<redacted>"[1](a088<redacted>) # "tj01-4090-node006 mlx5_2" lid 10 2xNDR
[4] "H-946d<redacted>"[1](946d<redacted>) # "tj01-4090-node007 mlx5_2" lid 12 2xNDR
[6] "H-946d<redacted>"[1](946d<redacted>) # "tj01-4090-node009 mlx5_2" lid 11 2xNDR
...omitted

Make sure perftest is installed.

On one node:

1

root@tj01-4090-node099:/# ib_send_bw --report_gbit -a -F -d mlx5_2

--report_gbit: Show result in gigabit.
-a: Run sizes from 2 till 2^23.
-F: Do not show a warning even if cpufreq_ondemand module is loaded, and cpu-freq is not on max.
-d: Use IB device. To make sure you are using the device you want to test (if you have multiple IB devices like me).

On another node:

1

root@tj01-h20-node139:/# ib_send_bw --report_gbit -a -F -d mlx5_2 tj01-4090-node099

tj01-4090-node099 is the IP address (hostname) of the other node.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


---------------------------------------------------------------------------------------
 Send BW Test
 Dual-port : OFF Device : mlx5_2
 Number of qps : 1 Transport type : IB
 Connection type : RC Using SRQ : OFF
 PCIe relax order: ON
 ibv_wr* API : ON
 TX depth : 128
 CQ Moderation : 100
 Mtu : 4096[B]
 Link type : IB
 Max inline data : 0[B]
 rdma_cm QPs : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x6c QPN 0x0050 PSN 0xddd611
 remote address: LID 0x28 QPN 0x0050 PSN 0xa3c746
---------------------------------------------------------------------------------------
 #bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
 2 1000 0.052812 0.050794 3.174618
 4 1000 0.080899 0.080547 2.517093
 8 1000 0.17 0.17 2.624748
 16 1000 0.35 0.35 2.741373
 32 1000 0.70 0.70 2.736427
 64 1000 1.62 1.62 3.159528
 128 1000 3.07 3.07 2.994503
 256 1000 5.46 5.46 2.663988
 512 1000 13.24 13.23 3.230121
 1024 1000 26.58 26.56 3.241834
 2048 1000 51.80 51.75 3.158744
 4096 1000 82.15 82.11 2.505647
 8192 1000 89.37 89.33 1.363014
 16384 1000 89.01 87.24 0.665597
 32768 1000 172.67 96.54 0.368268
 65536 1000 181.95 108.83 0.207586
 131072 1000 181.82 137.78 0.131399
 262144 1000 187.32 163.15 0.077797
 524288 1000 190.12 179.08 0.042696
 1048576 1000 191.18 187.28 0.022326
 2097152 1000 192.69 192.69 0.011485
 4194304 1000 195.53 195.53 0.005827
 8388608 1000 196.69 196.69 0.002931
---------------------------------------------------------------------------------------