基于 Mellanox CX-6 网卡的 VF-LAG 实践-天翼云开发者社区

一、技术简介

VF-LAG（Virtual Function Link Aggregation Group）是 Mellanox/NVIDIA 网卡（如 ConnectX-5/6/7）支持的一种硬件级链路聚合技术。它允许将多端口物理网卡的 SR-IOV 虚拟功能（VF）跨端口聚合到同一个 LAG 组，实现高可用和LB，提升虚拟化环境下的网络性能和可靠性。

VF-LAG 相较于传统 SR-IOV，主要有以下优势：

高可用性（HA）

传统 SR-IOV：每个 VF 只能绑定在单一物理端口（PF），如果该端口链路断开，VF 网络中断。
VF-LAG：单个 VF 绑定在 LAG（多个物理端口聚合）上，任一物理端口断开，VF 仍可通过其他端口通信，实现链路冗余和无缝切换。

传统 SR-IOV：VF 的流量只能走单一物理端口，如果该端口链路断开，VF 网络中断。
VF-LAG：VF 的流量可在多个物理端口间分担，实现流量的均衡。

简化网络配置

传统 SR-IOV：需要在主机侧做 Bonding 或 Teaming，配置复杂，且 SR-IOV VF 直通场景下主机 Bonding 无法直接作用于 VF。
VF-LAG：LAG 聚合在硬件/固件层实现，VF 直通给虚拟机/容器后，VM/容器侧无需做 Bonding，网络高可用和 LB 自动生效。

更好的虚拟化支持

传统 SR-IOV：VF 直通后，虚拟机/容器侧无法感知和利用多端口聚合。
VF-LAG：VF 直通后，VM/容器自动获得 LAG 能力，无需额外配置，适合云原生和高性能虚拟化场景。

简化运维

传统 SR-IOV：链路聚合、故障切换等需主机和虚拟机多层配合，运维复杂。
VF-LAG：所有聚合和切换在硬件完成，极大简化网络运维。

因此，可以看出VF-LAG 让 SR-IOV VF 具备了硬件级 LAG 的高可用和LB能力，极大提升了虚拟化和云原生场景下的网络可靠性与性能，同时简化了运维。

二、原理简述

传统 SR-IOV：每个 VF 只能绑定在单一物理端口。
VF-LAG：多个物理端口组成 LAG，VF 绑定在 LAG 上，流量可在多个端口间分担。
VF-LAG 由固件和驱动协同实现，DPDK 侧无需特殊适配。

三、配置步骤

3.1 前置条件

需要有支持 VF-LAG 的 Mellanox 双端口网卡（如 ConnectX-5/6/7），VF-LAG 不支持跨网卡，需要使用同一张网卡的 2 个端口组建 BOND。

# lshw -c network -businfo
Bus info          Device        Class          Description
==========================================================
pci@0000:01:00.0  enp1s0f0np0   network        MT2892 Family [ConnectX-6 Dx]
pci@0000:01:00.1  enp1s0f1np1   network        MT2892 Family [ConnectX-6 Dx]

3.2 配置步骤

启用 sriov 功能

# echo 4 > /sys/class/net/enp1s0f0np0/device/sriov_numvfs
# echo 4 > /sys/class/net/enp1s0f1np1/device/sriov_numvfs
# ip link show enp1s0f0np0
692: enp1s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 94:6d:ae:8f:c1:c2 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
# ip link show enp1s0f1np1
693: enp1s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 94:6d:ae:8f:c1:c3 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off

解绑 vf 驱动

# lshw -c network -businfo
Bus info          Device        Class          Description
==========================================================
pci@0000:01:00.0  enp1s0f0np0   network        MT2892 Family [ConnectX-6 Dx]
pci@0000:01:00.1  enp1s0f1np1   network        MT2892 Family [ConnectX-6 Dx]
pci@0000:01:00.2  enp1s0f0v0    network        ConnectX Family mlx5Gen Virtual Function
pci@0000:01:00.3  enp1s0f0v1    network        ConnectX Family mlx5Gen Virtual Function
pci@0000:01:00.4  enp1s0f0v2    network        ConnectX Family mlx5Gen Virtual Function
pci@0000:01:00.5  enp1s0f0v3    network        ConnectX Family mlx5Gen Virtual Function
pci@0000:01:04.2  enp1s0f1v0    network        ConnectX Family mlx5Gen Virtual Function
pci@0000:01:04.3  enp1s0f1v1    network        ConnectX Family mlx5Gen Virtual Function
pci@0000:01:04.4  enp1s0f1v2    network        ConnectX Family mlx5Gen Virtual Function
pci@0000:01:04.5  enp1s0f1v3    network        ConnectX Family mlx5Gen Virtual Function

# echo 0000:01:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind
# echo 0000:01:00.3 > /sys/bus/pci/drivers/mlx5_core/unbind
# echo 0000:01:00.4 > /sys/bus/pci/drivers/mlx5_core/unbind
# echo 0000:01:00.5 > /sys/bus/pci/drivers/mlx5_core/unbind
# echo 0000:01:04.2 > /sys/bus/pci/drivers/mlx5_core/unbind
# echo 0000:01:04.3 > /sys/bus/pci/drivers/mlx5_core/unbind
# echo 0000:01:04.4 > /sys/bus/pci/drivers/mlx5_core/unbind
# echo 0000:01:04.5 > /sys/bus/pci/drivers/mlx5_core/unbind

配置网卡 switchdev模式 && 开启硬件offload

VF LAG 需要将 LAG 功能 offload 到硬件 eswitch，并且需要下发对应的匹配规则，因此需要网卡配置 switchdev 模式，并开启网卡硬件offload功能。

# devlink dev eswitch set pci/0000:01:00.0 mode switchdev
# devlink dev eswitch set pci/0000:01:00.1 mode switchdev

# devlink dev eswitch show pci/0000:01:00.0
pci/0000:01:00.0: mode switchdev inline-mode none encap-mode basic
# devlink dev eswitch show pci/0000:01:00.1
pci/0000:01:00.1: mode switchdev inline-mode none encap-mode basic

# ethtool -K enp1s0f0np0 hw-tc-offload on
# ethtool -K enp1s0f1np1 hw-tc-offload on

# ethtool -k enp1s0f0np0 | grep hw-tc-offload
hw-tc-offload: on
# ethtool -k enp1s0f1np1 | grep hw-tc-offload
hw-tc-offload: on

配置 BOND

vf lag 需要组建硬件的 bond，该bond 只能使用同一张物理网卡的两个物理口组建 bond，不可以跨网卡。支持的 bond 模式有三种：Active-backup、XOR 和 LACP。使用硬件offload功能之后，通过下发规则，使两个 pf 口的数据包可转发到任何一个 pf 下的 vf 口。对于 vf 口出向的流量在主备 bond 模式下，只有一个 pf 运行，VF 的流量只会通过该 pf 口；在 XOR 或 LACP bond 模式下，如果两个 PF 均正常运行，则 vf 出向的流量会在这两个 pf 之间进行流量均衡。

以下使用常用的 LACP bond 的模式为例：

# modprobe bonding mode=802.3ad
# ip link add vf-bond type bond mode 802.3ad
# ip link set enp1s0f0np0 down
# ip link set enp1s0f1np1 down
# ip link set enp1s0f0np0 master vf-bond
# ip link set enp1s0f1np1 master vf-bond
# ip link set enp1s0f0np0 up
# ip link set enp1s0f1np1 up

# mst status -v
MST modules:
------------
    MST PCI module is not loaded
    MST PCI configuration module loaded
PCI devices:
------------
DEVICE_TYPE             MST                           PCI       RDMA            NET                                     NUMA
ConnectX6DX(rev:0)      /dev/mst/mt4125_pciconf0.1    01:00.1   mlx5_bond_0     net-enp1s0f1npf1vf0,net-enp1s0f1npf1vf3,net-vf-bond,0

ConnectX6DX(rev:0)      /dev/mst/mt4125_pciconf0      01:00.0   mlx5_bond_0     net-enp1s0f0npf0vf2,net-vf-bond,net-enp1s0f0npf0vf0,0

# dmest -T
...
[四 6月 26 18:07:45 2025] mlx5_core 0000:01:00.0 enp1s0f0np0: Link down
[四 6月 26 18:07:45 2025] vf-bond: (slave enp1s0f0np0): Enslaving as a backup interface with a down link
[四 6月 26 18:07:52 2025] mlx5_core 0000:01:00.1 enp1s0f1np1: Link down
[四 6月 26 18:07:52 2025] vf-bond: (slave enp1s0f1np1): Enslaving as a backup interface with a down link
[四 6月 26 18:07:52 2025] mlx5_core 0000:01:00.0 enp1s0f0np0: Link up
[四 6月 26 18:07:52 2025] mlx5_core 0000:01:00.0: lag map active ports: 1, 2
[四 6月 26 18:07:52 2025] mlx5_core 0000:01:00.0: shared_fdb:1 mode:hash
[四 6月 26 18:07:52 2025] mlx5_core 0000:01:00.1 enp1s0f1np1: Link down
[四 6月 26 18:07:52 2025] mlx5_core 0000:01:00.0: Operation mode is single FDB
[四 6月 26 18:07:59 2025] mlx5_core 0000:01:00.1 enp1s0f1np1: Link up
[四 6月 26 18:07:59 2025] mlx5_core 0000:01:00.0 enp1s0f0np0: Link up
...

重新绑定 vf 驱动

# echo 0000:01:00.2 > /sys/bus/pci/drivers/mlx5_core/bind
# echo 0000:01:00.3 > /sys/bus/pci/drivers/mlx5_core/bind
# echo 0000:01:00.4 > /sys/bus/pci/drivers/mlx5_core/bind
# echo 0000:01:00.5 > /sys/bus/pci/drivers/mlx5_core/bind
# echo 0000:01:04.2 > /sys/bus/pci/drivers/mlx5_core/bind
# echo 0000:01:04.3 > /sys/bus/pci/drivers/mlx5_core/bind
# echo 0000:01:04.4 > /sys/bus/pci/drivers/mlx5_core/bind
# echo 0000:01:04.5 > /sys/bus/pci/drivers/mlx5_core/bind

下发规则

下面简单下发一条入向的规则到两个 pf 口作为测试使用，该规则匹配目的 vf mac，将流量均转发到该 vf representor port。

# ethtool -i enp1s0f0npf0vf0
driver: mlx5e_rep
version: 5.10.0-136.12.0.88.4.ctl3.x86_6
firmware-version: 22.43.2026 (MT_0000000359)
expansion-rom-version:
bus-info:
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

# ethtool -i enp1s0f0v0
driver: mlx5_core
version: 24.10-1.1.4
firmware-version: 22.43.2026 (MT_0000000359)
expansion-rom-version:
bus-info: 0000:01:00.2
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

# ethtool -K enp1s0f0npf0vf0 hw-tc-offload on
# tc qdisc add dev vf-bond ingress_block 22 ingress
# tc qdisc add dev enp1s0f0np0 ingress_block 22 ingress
# tc qdisc add dev enp1s0f1np1 ingress_block 22 ingress
# tc filter add block 22 protocol 0x0800 parent ffff: prio 1 flower dst_mac 52:54:00:00:00:01 action mirred egress redirect dev enp1s0f0npf0vf0

testpmd 接管 vf

交换机侧需要配置 lacp bond，对端打多流，这里略过。

# dpdk-testpmd -l 1-2 -n 4 -a 0000:01:00.2 -- -i
testpmd> mac_addr set 0 52:54:00:00:00:01
testpmd> set verbose 1
Change verbose level from 0 to 1
testpmd> start

打流的过程中，可以发现 vf 口可以抓到对应的报文。在交换机侧通过切换对应的 PF 口链路状态，可以看到 vf 口均可以正常收到报文。

# lshw -c network -businfo Bus info Device Class Description ========================================================== pci@0000:01:00.0 enp1s0f0np0 network MT2892 Family [ConnectX-6 Dx] pci@0000:01:00.1 enp1s0f1np1 network MT2892 Family [ConnectX-6 Dx]

# echo 4 > /sys/class/net/enp1s0f0np0/device/sriov_numvfs # echo 4 > /sys/class/net/enp1s0f1np1/device/sriov_numvfs # ip link show enp1s0f0np0 692: enp1s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 94:6d:ae:8f:c1:c2 brd ff:ff:ff:ff:ff:ff vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off # ip link show enp1s0f1np1 693: enp1s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 94:6d:ae:8f:c1:c3 brd ff:ff:ff:ff:ff:ff vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off

# lshw -c network -businfo Bus info Device Class Description ========================================================== pci@0000:01:00.0 enp1s0f0np0 network MT2892 Family [ConnectX-6 Dx] pci@0000:01:00.1 enp1s0f1np1 network MT2892 Family [ConnectX-6 Dx] pci@0000:01:00.2 enp1s0f0v0 network ConnectX Family mlx5Gen Virtual Function pci@0000:01:00.3 enp1s0f0v1 network ConnectX Family mlx5Gen Virtual Function pci@0000:01:00.4 enp1s0f0v2 network ConnectX Family mlx5Gen Virtual Function pci@0000:01:00.5 enp1s0f0v3 network ConnectX Family mlx5Gen Virtual Function pci@0000:01:04.2 enp1s0f1v0 network ConnectX Family mlx5Gen Virtual Function pci@0000:01:04.3 enp1s0f1v1 network ConnectX Family mlx5Gen Virtual Function pci@0000:01:04.4 enp1s0f1v2 network ConnectX Family mlx5Gen Virtual Function pci@0000:01:04.5 enp1s0f1v3 network ConnectX Family mlx5Gen Virtual Function # echo 0000:01:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind # echo 0000:01:00.3 > /sys/bus/pci/drivers/mlx5_core/unbind # echo 0000:01:00.4 > /sys/bus/pci/drivers/mlx5_core/unbind # echo 0000:01:00.5 > /sys/bus/pci/drivers/mlx5_core/unbind # echo 0000:01:04.2 > /sys/bus/pci/drivers/mlx5_core/unbind # echo 0000:01:04.3 > /sys/bus/pci/drivers/mlx5_core/unbind # echo 0000:01:04.4 > /sys/bus/pci/drivers/mlx5_core/unbind # echo 0000:01:04.5 > /sys/bus/pci/drivers/mlx5_core/unbind

# devlink dev eswitch set pci/0000:01:00.0 mode switchdev # devlink dev eswitch set pci/0000:01:00.1 mode switchdev # devlink dev eswitch show pci/0000:01:00.0 pci/0000:01:00.0: mode switchdev inline-mode none encap-mode basic # devlink dev eswitch show pci/0000:01:00.1 pci/0000:01:00.1: mode switchdev inline-mode none encap-mode basic # ethtool -K enp1s0f0np0 hw-tc-offload on # ethtool -K enp1s0f1np1 hw-tc-offload on # ethtool -k enp1s0f0np0 | grep hw-tc-offload hw-tc-offload: on # ethtool -k enp1s0f1np1 | grep hw-tc-offload hw-tc-offload: on

# modprobe bonding mode=802.3ad # ip link add vf-bond type bond mode 802.3ad # ip link set enp1s0f0np0 down # ip link set enp1s0f1np1 down # ip link set enp1s0f0np0 master vf-bond # ip link set enp1s0f1np1 master vf-bond # ip link set enp1s0f0np0 up # ip link set enp1s0f1np1 up

# mst status -v MST modules: ------------ MST PCI module is not loaded MST PCI configuration module loaded PCI devices: ------------ DEVICE_TYPE MST PCI RDMA NET NUMA ConnectX6DX(rev:0) /dev/mst/mt4125_pciconf0.1 01:00.1 mlx5_bond_0 net-enp1s0f1npf1vf0,net-enp1s0f1npf1vf3,net-vf-bond,0 ConnectX6DX(rev:0) /dev/mst/mt4125_pciconf0 01:00.0 mlx5_bond_0 net-enp1s0f0npf0vf2,net-vf-bond,net-enp1s0f0npf0vf0,0 # dmest -T ... [四 6月 26 18:07:45 2025] mlx5_core 0000:01:00.0 enp1s0f0np0: Link down [四 6月 26 18:07:45 2025] vf-bond: (slave enp1s0f0np0): Enslaving as a backup interface with a down link [四 6月 26 18:07:52 2025] mlx5_core 0000:01:00.1 enp1s0f1np1: Link down [四 6月 26 18:07:52 2025] vf-bond: (slave enp1s0f1np1): Enslaving as a backup interface with a down link [四 6月 26 18:07:52 2025] mlx5_core 0000:01:00.0 enp1s0f0np0: Link up [四 6月 26 18:07:52 2025] mlx5_core 0000:01:00.0: lag map active ports: 1, 2 [四 6月 26 18:07:52 2025] mlx5_core 0000:01:00.0: shared_fdb:1 mode:hash [四 6月 26 18:07:52 2025] mlx5_core 0000:01:00.1 enp1s0f1np1: Link down [四 6月 26 18:07:52 2025] mlx5_core 0000:01:00.0: Operation mode is single FDB [四 6月 26 18:07:59 2025] mlx5_core 0000:01:00.1 enp1s0f1np1: Link up [四 6月 26 18:07:59 2025] mlx5_core 0000:01:00.0 enp1s0f0np0: Link up ...

# echo 0000:01:00.2 > /sys/bus/pci/drivers/mlx5_core/bind # echo 0000:01:00.3 > /sys/bus/pci/drivers/mlx5_core/bind # echo 0000:01:00.4 > /sys/bus/pci/drivers/mlx5_core/bind # echo 0000:01:00.5 > /sys/bus/pci/drivers/mlx5_core/bind # echo 0000:01:04.2 > /sys/bus/pci/drivers/mlx5_core/bind # echo 0000:01:04.3 > /sys/bus/pci/drivers/mlx5_core/bind # echo 0000:01:04.4 > /sys/bus/pci/drivers/mlx5_core/bind # echo 0000:01:04.5 > /sys/bus/pci/drivers/mlx5_core/bind

# ethtool -i enp1s0f0npf0vf0 driver: mlx5e_rep version: 5.10.0-136.12.0.88.4.ctl3.x86_6 firmware-version: 22.43.2026 (MT_0000000359) expansion-rom-version: bus-info: supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no # ethtool -i enp1s0f0v0 driver: mlx5_core version: 24.10-1.1.4 firmware-version: 22.43.2026 (MT_0000000359) expansion-rom-version: bus-info: 0000:01:00.2 supports-statistics: yes supports-test: yes supports-eeprom-access: no supports-register-dump: no supports-priv-flags: yes # ethtool -K enp1s0f0npf0vf0 hw-tc-offload on # tc qdisc add dev vf-bond ingress_block 22 ingress # tc qdisc add dev enp1s0f0np0 ingress_block 22 ingress # tc qdisc add dev enp1s0f1np1 ingress_block 22 ingress # tc filter add block 22 protocol 0x0800 parent ffff: prio 1 flower dst_mac 52:54:00:00:00:01 action mirred egress redirect dev enp1s0f0npf0vf0

智算服务

应用商城

合作伙伴

开发者

支持与服务

了解天翼云

基于 Mellanox CX-6 网卡的 VF-LAG 实践

基于 Mellanox CX-6 网卡的 VF-LAG 实践

活动

智算服务

应用商城

合作伙伴

开发者

支持与服务

了解天翼云

基于 Mellanox CX-6 网卡的 VF-LAG 实践

基于 Mellanox CX-6 网卡的 VF-LAG 实践