MTU配置不当问题-天翼云开发者社区

现象

在主机内能 ping 通互联网的地址，但是从互联网拉取文件时一直超时。

排查过程

网络拓扑为：互联网 <-> 逻辑网关 <-> 主机，即主机访问互联网时需要经过逻辑网关，逻辑网关里配置了用于访问互联网的NAT相关规则。

在主机内使用 curl 或 wget 时，增加日志级别，可以观察到连接已经建立了，但是后续却没有数据，直到超时退出。

由于出网经过逻辑网关，因此在逻辑网关内抓包观察，以下是分别在逻辑网关两侧的抓包结果，先看互联网侧的抓包结果，如下所示：

15:18:12.793219 IP 192.168.10.39.36302 > 116.253.29.205.443: Flags [S], seq 2488013264, win 65535, options [mss 1460,sackOK,TS val 4141372887 ecr 0,nop,wscale 8], length 0
15:18:12.821648 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [S.], seq 3457463034, ack 2488013265, win 42340, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0
15:18:12.822230 IP 192.168.10.39.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, length 0
15:18:12.910971 IP 192.168.10.39.36302 > 116.253.29.205.443: Flags [P.], seq 1:196, ack 1, win 256, length 195
15:18:12.939368 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], ack 196, win 83, length 0

15:18:12.946179 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:18:12.946198 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1461:2921, ack 196, win 83, length 1460
15:18:12.949090 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [P.], seq 2921:4374, ack 196, win 83, length 1453
15:18:13.009258 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [P.], seq 2921:4374, ack 196, win 83, length 1453
15:18:13.240253 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:18:13.704255 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:18:14.672258 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:18:16.528279 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:18:20.240243 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:18:27.856268 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:18:42.704333 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460

15:18:42.850948 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [F.], seq 4374, ack 196, win 83, length 0
15:18:42.852265 IP 192.168.10.39.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0
15:18:42.880688 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:19:43.032513 IP 192.168.10.39.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0
15:19:43.060908 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [R], seq 3457463035, win 0, length 0

其中，192.168.10.39 是 NAT 转换后的地址，从上面可以看到 1-5 行是三次握手，建立连接的日志；第 7-17 行是服务器发送数据的日志，其中可以看到 1:1461 的包一直在重发（因为没有收到响应）；19-23 行是服务器通知逻辑网关关闭连接的日志。

再看内网侧的抓包结果，如下所示：

15:18:12.793177 IP 10.0.0.6.36302 > 116.253.29.205.443: Flags [S], seq 2488013264, win 65535, options [mss 1460,sackOK,TS val 4141372887 ecr 0,nop,wscale 8], length 0
15:18:12.821677 IP 116.253.29.205.443 > 10.0.0.6.36302: Flags [S.], seq 3457463034, ack 2488013265, win 42340, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0
15:18:12.822207 IP 10.0.0.6.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, length 0
15:18:12.910939 IP 10.0.0.6.36302 > 116.253.29.205.443: Flags [P.], seq 1:196, ack 1, win 256, length 195
15:18:12.939388 IP 116.253.29.205.443 > 10.0.0.6.36302: Flags [.], ack 196, win 83, length 0

15:18:42.850974 IP 116.253.29.205.443 > 10.0.0.6.36302: Flags [F.], seq 4374, ack 196, win 83, length 0
15:18:42.852238 IP 10.0.0.6.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0
15:19:43.032468 IP 10.0.0.6.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0
15:19:43.060929 IP 116.253.29.205.443 > 10.0.0.6.36302: Flags [R], seq 3457463035, win 0, length 0

其中，10.0.0.6 是 NAT 转换前的地址，1-5 行也是建立连接的日志，然后 7-10 行就已经是关闭连接的日志了。和前面的抓包结果对比可以很直观的发现，服务器发送过来的数据包，内部网卡都没有收到，导致最后服务器关闭了连接。

那么现在的问题就是为什么内部网卡会收不到这些数据包？

重新看日志的时候发现，建立连接的时候，双方约定了mss（最大报文段长度）为1460 （对应日志 [mss 1460,sackOK,TS val 4141372887 ecr 0,nop,wscale 8]），说明主机内的网卡MTU（最大传输单元）应该是 1500 字节，但是，逻辑网关的网卡 MTU 是 1400 ，这就导致了如果收到的报文长度是大于 1400 字节的就会被丢弃，这和前面观察到的现象基本一致，并且服务器发来的报文长度就是1460（seq 1:1461）。

由于前面抓包的时候限制了协议，这里限制 host 后再抓包看下结果：

tcpdump -nn -i net1 host 116.253.29.205

15:38:11.303605 IP 192.168.10.39.36306 > 116.253.29.205.443: Flags [S], seq 2941142219, win 65535, options [mss 1460,sackOK,TS val 4142571398 ecr 0,nop,wscale 8], length 0
15:38:11.320427 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [S.], seq 2764254742, ack 2941142220, win 42340, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0
15:38:11.320981 IP 192.168.10.39.36306 > 116.253.29.205.443: Flags [.], ack 1, win 256, length 0
15:38:11.411741 IP 192.168.10.39.36306 > 116.253.29.205.443: Flags [P.], seq 1:196, ack 1, win 256, length 195
15:38:11.428650 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], ack 196, win 83, length 0
15:38:11.431748 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:11.431785 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:11.431764 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1461:2921, ack 196, win 83, length 1460
15:38:11.431812 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:11.433477 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [P.], seq 2921:4374, ack 196, win 83, length 1453
15:38:11.433488 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:11.470625 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [P.], seq 2921:4374, ack 196, win 83, length 1453
15:38:11.470662 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:11.694652 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:11.694692 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:12.134651 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:12.134690 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:13.014626 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:13.014662 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:14.806622 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:14.806661 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:18.326657 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:18.326696 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:25.366616 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:25.366658 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:39.702639 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:39.702686 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:41.337719 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [F.], seq 4374, ack 196, win 83, length 0
15:38:41.338581 IP 192.168.10.39.36306 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0
15:38:41.355434 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:41.355472 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:39:41.624210 IP 192.168.10.39.36306 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0
15:39:41.641021 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [R], seq 2764254743, win 0, length 0

这次的抓包结果就能说明确实是因为逻辑网关网卡的 MTU 为 1400 导致的。

解决方法

知道原因后，解决方法基本就清晰了，在报文经过逻辑网关时修改 MSS 的值即可，使其根据路径MTU中的最小值进行计算，这里有个专业的术语叫 **MSS钳制**，具体的操作指令如下所示。

iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

排查过程

网络拓扑为：互联网 <-> 逻辑网关 <-> 主机，即主机访问互联网时需要经过逻辑网关，逻辑网关里配置了用于访问互联网的NAT相关规则。

在主机内使用 curl 或 wget 时，增加日志级别，可以观察到连接已经建立了，但是后续却没有数据，直到超时退出。

由于出网经过逻辑网关，因此在逻辑网关内抓包观察，以下是分别在逻辑网关两侧的抓包结果，先看互联网侧的抓包结果，如下所示：

15:18:12.793219 IP 192.168.10.39.36302 > 116.253.29.205.443: Flags [S], seq 2488013264, win 65535, options [mss 1460,sackOK,TS val 4141372887 ecr 0,nop,wscale 8], length 0 15:18:12.821648 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [S.], seq 3457463034, ack 2488013265, win 42340, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0 15:18:12.822230 IP 192.168.10.39.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, length 0 15:18:12.910971 IP 192.168.10.39.36302 > 116.253.29.205.443: Flags [P.], seq 1:196, ack 1, win 256, length 195 15:18:12.939368 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], ack 196, win 83, length 0 15:18:12.946179 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:18:12.946198 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1461:2921, ack 196, win 83, length 1460 15:18:12.949090 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [P.], seq 2921:4374, ack 196, win 83, length 1453 15:18:13.009258 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [P.], seq 2921:4374, ack 196, win 83, length 1453 15:18:13.240253 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:18:13.704255 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:18:14.672258 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:18:16.528279 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:18:20.240243 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:18:27.856268 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:18:42.704333 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:18:42.850948 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [F.], seq 4374, ack 196, win 83, length 0 15:18:42.852265 IP 192.168.10.39.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0 15:18:42.880688 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:19:43.032513 IP 192.168.10.39.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0 15:19:43.060908 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [R], seq 3457463035, win 0, length 0

再看内网侧的抓包结果，如下所示：

15:18:12.793177 IP 10.0.0.6.36302 > 116.253.29.205.443: Flags [S], seq 2488013264, win 65535, options [mss 1460,sackOK,TS val 4141372887 ecr 0,nop,wscale 8], length 0 15:18:12.821677 IP 116.253.29.205.443 > 10.0.0.6.36302: Flags [S.], seq 3457463034, ack 2488013265, win 42340, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0 15:18:12.822207 IP 10.0.0.6.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, length 0 15:18:12.910939 IP 10.0.0.6.36302 > 116.253.29.205.443: Flags [P.], seq 1:196, ack 1, win 256, length 195 15:18:12.939388 IP 116.253.29.205.443 > 10.0.0.6.36302: Flags [.], ack 196, win 83, length 0 15:18:42.850974 IP 116.253.29.205.443 > 10.0.0.6.36302: Flags [F.], seq 4374, ack 196, win 83, length 0 15:18:42.852238 IP 10.0.0.6.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0 15:19:43.032468 IP 10.0.0.6.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0 15:19:43.060929 IP 116.253.29.205.443 > 10.0.0.6.36302: Flags [R], seq 3457463035, win 0, length 0

那么现在的问题就是为什么内部网卡会收不到这些数据包？

由于前面抓包的时候限制了协议，这里限制 host 后再抓包看下结果：

tcpdump -nn -i net1 host 116.253.29.205 15:38:11.303605 IP 192.168.10.39.36306 > 116.253.29.205.443: Flags [S], seq 2941142219, win 65535, options [mss 1460,sackOK,TS val 4142571398 ecr 0,nop,wscale 8], length 0 15:38:11.320427 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [S.], seq 2764254742, ack 2941142220, win 42340, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0 15:38:11.320981 IP 192.168.10.39.36306 > 116.253.29.205.443: Flags [.], ack 1, win 256, length 0 15:38:11.411741 IP 192.168.10.39.36306 > 116.253.29.205.443: Flags [P.], seq 1:196, ack 1, win 256, length 195 15:38:11.428650 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], ack 196, win 83, length 0 15:38:11.431748 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:38:11.431785 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556 15:38:11.431764 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1461:2921, ack 196, win 83, length 1460 15:38:11.431812 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556 15:38:11.433477 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [P.], seq 2921:4374, ack 196, win 83, length 1453 15:38:11.433488 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556 15:38:11.470625 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [P.], seq 2921:4374, ack 196, win 83, length 1453 15:38:11.470662 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556 15:38:11.694652 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:38:11.694692 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556 15:38:12.134651 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:38:12.134690 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556 15:38:13.014626 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:38:13.014662 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556 15:38:14.806622 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:38:14.806661 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556 15:38:18.326657 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:38:18.326696 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556 15:38:25.366616 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:38:25.366658 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556 15:38:39.702639 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:38:39.702686 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556 15:38:41.337719 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [F.], seq 4374, ack 196, win 83, length 0 15:38:41.338581 IP 192.168.10.39.36306 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0 15:38:41.355434 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460 15:38:41.355472 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556 15:39:41.624210 IP 192.168.10.39.36306 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0 15:39:41.641021 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [R], seq 2764254743, win 0, length 0

这次的抓包结果就能说明确实是因为逻辑网关网卡的 MTU 为 1400 导致的。

解决方法

iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

智算服务

应用商城

合作伙伴

开发者

支持与服务

了解天翼云

MTU配置不当问题

现象

排查过程

解决方法

MTU配置不当问题

现象

排查过程

解决方法

活动

智算服务

应用商城

合作伙伴

开发者

支持与服务

了解天翼云

MTU配置不当问题

现象

排查过程

解决方法

MTU配置不当问题

现象

排查过程

解决方法