Hacking on Charlie Chiang's bloghttps://charlie0129.github.io/blog/categories/hacking/Recent content in Hacking on Charlie Chiang's blogHugo -- gohugo.ioenFri, 19 Jul 2024 16:37:00 +0800Flash 10G CX4121A to 25Ghttps://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/Fri, 19 Jul 2024 16:37:00 +0800https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/<p>10G ConnectX-4 Lx NICs can be flashed to 25G with the right firmware. This is useful if you have a 10G NIC and want to upgrade to 25G without buying a new NIC. This article will guide you through the process of flashing a 10G Mellanox ConnectX-4 Lx NIC to 25G.</p> <p><img src="https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-10g.jpg" width="4032" height="3024" srcset="https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-10g_hu_e67c9199f79997b.webp 480w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-10g_hu_f473233eaea7608d.jpg 480w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-10g_hu_c3aef0b072ec866.webp 1024w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-10g_hu_dc742250568c7fee.jpg 1024w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-10g_hu_f66edbb97499b063.webp 1536w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-10g_hu_78d6e0bb25964f66.jpg 1536w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-10g_hu_91d79469bdb58ec7.webp 2048w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-10g_hu_30a548030205647.jpg 2048w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-10g_hu_4a93a94a834c4b11.webp 2560w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-10g_hu_da6e88a334cf467c.jpg 2560w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-10g_hu_be28c42d38efe826.webp 4032w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-10g_hu_b51529c87b4602cd.jpg 4032w" loading="lazy" alt="10G ConnectX-4 Lx" class="gallery-image" data-flex-grow="133" data-flex-basis="320px" > </p> <p><img src="https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-25g.jpg" width="3455" height="2591" srcset="https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-25g_hu_56e8aacbea38e6dd.webp 480w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-25g_hu_8e6b89cef8aee16d.jpg 480w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-25g_hu_ed897d5a8fd11782.webp 1024w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-25g_hu_25c0489be6304c6c.jpg 1024w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-25g_hu_7763f81dd3853a26.webp 1536w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-25g_hu_a70cb4540c546c23.jpg 1536w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-25g_hu_39f701fb24c38667.webp 2048w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-25g_hu_d0e4c3fbcae55987.jpg 2048w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-25g_hu_244c2634c4e6d6e.webp 2560w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-25g_hu_2b581b867a6ad0ea.jpg 2560w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-25g_hu_fa581e091ddafd5b.webp 3455w, https://charlie0129.github.io/blog/p/mcx4121a-10g-to-25g/images/cx4121a-25g_hu_581adf59df5a6c23.jpg 3455w" loading="lazy" alt="25G ConnectX-4 Lx" class="gallery-image" data-flex-grow="133" data-flex-basis="320px" > </p> <p>Warning: Do not try this on MCX4121C. It will brick the NIC.</p> <ol> <li> <p>Download firmware from <a class="link" href="https://network.nvidia.com/support/firmware/connectx4lxen/" target="_blank" rel="noopener" >Firmware for ConnectX®-4 Lx EN</a>. Choose <code>MCX4121A-ACUT</code>. I will assume the firmware file is named <code>fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACU_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin</code>.</p> <p>If you are wondering what&rsquo;s the difference between models, here&rsquo;s a table for ConnectX-4 Lx cards (<a class="link" href="files/pb-connectx-4-lx-en-card.pdf" >original document</a>). You should choose 25G models to flash to.</p> <table> <thead> <tr> <th>Max Network Speed</th> <th>Interface Type</th> <th>Supported Ethernet Speeds (GbE)</th> <th>Host Interface</th> <th>Additional Features</th> <th>OPN</th> </tr> </thead> <tbody> <tr> <td>1x 10GbE</td> <td>SFP28</td> <td>10, 1</td> <td>PCIe 3.0 x8</td> <td></td> <td>MCX4111A-XCAT</td> </tr> <tr> <td>2x 10GbE</td> <td>SFP28</td> <td>10, 1</td> <td>PCIe 3.0 x8</td> <td></td> <td>MCX4121A-XCAT</td> </tr> <tr> <td>2x 10GbE</td> <td>SFP28</td> <td>10, 1</td> <td>PCIe 3.0 x8</td> <td>Host Management, UEFI Enabled</td> <td>MCX4121A-XCHT</td> </tr> <tr> <td>1x 25GbE</td> <td>SFP28</td> <td>25, 10, 1</td> <td>PCIe 3.0 x8</td> <td></td> <td>MCX4111A-ACAT</td> </tr> <tr> <td>1x 25GbE</td> <td>SFP28</td> <td>25, 10, 1</td> <td>PCIe 3.0 x8</td> <td>UEFI Enabled</td> <td>MCX4111A-ACUT</td> </tr> <tr> <td>2x 25GbE</td> <td>SFP28</td> <td>25, 10, 1</td> <td>PCIe 3.0 x8</td> <td></td> <td>MCX4121A-ACAT</td> </tr> <tr> <td>2x 25GbE</td> <td>SFP28</td> <td>25, 10, 1</td> <td>PCIe 3.0 x8</td> <td>UEFI Enabled</td> <td>MCX4121A-ACUT</td> </tr> <tr> <td>1x 40GbE</td> <td>QSFP28</td> <td>40, 25, 10, 1</td> <td>PCIe 3.0 x8</td> <td></td> <td>MCX4131A-BCAT</td> </tr> <tr> <td>1x 50GbE</td> <td>QSFP28</td> <td>50, 40, 25, 10, 1</td> <td>PCIe 3.0 x8</td> <td></td> <td>MCX4131A-GCAT</td> </tr> </tbody> </table> </li> <li> <p>Download and install <a class="link" href="https://network.nvidia.com/products/adapter-software/firmware-tools/" target="_blank" rel="noopener" >NVIDIA Firmware Tools (MFT)</a>.</p> </li> <li> <p><code>mst start</code></p> </li> <li> <p><code>mst ststus</code> You will see you MST device, e.g. <code>/dev/mst/mt4117_pciconf0</code>. I will use this device path in this article. If your path differs from this, please change it accordingly.</p> </li> <li> <p>Save GUID, MAC: <code>flint -d /dev/mst/mt4117_pciconf0 query full &gt; flint_query.txt</code></p> </li> <li> <p>Save hardware info: <code>flint -d /dev/mst/mt4117_pciconf0 hw query &gt; flint_hwinfo.txt</code></p> </li> <li> <p>Save current firmware: <code>flint -d /dev/mst/mt4117_pciconf0 ri orig_firmware.bin</code></p> </li> <li> <p>Save current firmware config: <code>flint -d /dev/mst/mt4117_pciconf0 dc orig_firmware.ini</code></p> </li> <li> <p>Save current PXE ROM (if exists): <code>flint -d /dev/mst/mt4117_pciconf0 rrom orig_rom.bin</code></p> </li> <li> <p>Save current PCI VPD: <code>mlxburn -d /dev/mst/mt4117_pciconf0 -vpd &gt; orig_vpd.txt</code></p> </li> <li> <p><code>flint -i fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACU_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin verify</code></p> </li> <li> <p><code>flint -i fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACU_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin -d /dev/mst/mt4117_pciconf0 -allow_psid_change burn</code></p> </li> </ol> <p>Reboot you machine. You should see your NIC is now have 25G speed.</p> <p>Bonus:</p> <ul> <li>To change MAC address: <code>flint -d /dev/mst/mt4117_pciconf0 -mac 02c90abcdef0 sg</code></li> <li>To change GUID: <code>flint -d /dev/mst/mt4117_pciconf0 -guid 0002c9000abcdef0 sg</code></li> </ul>