实验环境如下图所示：
配置如下：
#!/bin/bash sudo ip netns add ns1 sudo ip link add ns1veth1 type veth peer name eth0 netns ns1 sudo ip netns add ns2 sudo ip link add ns2veth1 type veth peer name eth0 netns ns2 sudo ip link set ns1veth1 master vrftest sudo ip link set ns2veth1 master vrftest sudo ip link set ns2veth1 up sudo ip link set ns1veth1 up sudo ip addr add 1.1.1.254/24 dev ns1veth1 sudo ip addr add 2.2.2.254/24 dev ns2veth1 sudo ip netns exec ns2 ip addr add 2.2.2.1/24 dev eth0 sudo ip netns exec ns1 ip addr add 1.1.1.1/24 dev eth0 sudo ip netns exec ns1 ip link set eth0 up sudo ip netns exec ns1 ip link set lo up sudo ip netns exec ns1 ip route add default via 1.1.1.254 dev eth0 sudo ip netns exec ns2 ip link set eth0 up sudo ip netns exec ns2 ip link set lo up sudo ip netns exec ns2 ip route add default via 2.2.2.254 dev eth0 sudo iptables -t mangle -A PREROUTING -s 1.1.1.1 -j LOG --log-prefix="vrf-test-prerouting" sudo iptables -t mangle -A FORWARD -s 1.1.1.1 -j LOG --log-prefix="vrf-test-forward" sudo iptables -t mangle -A POSTROUTING -s 1.1.1.1 -j LOG --log-prefix="vrf-test-postrouting" sudo iptables -t mangle -A PREROUTING -d 1.1.1.1 -j LOG --log-prefix="vrf-test-prerouting" sudo iptables -t mangle -A FORWARD -d 1.1.1.1 -j LOG --log-prefix="vrf-test-forward" sudo iptables -t mangle -A POSTROUTING -d 1.1.1.1 -j LOG --log-prefix="vrf-test-postrouting" sudo iptables -t mangle -A INPUT -d 1.1.1.1 -j LOG --log-prefix="vrf-test-localin" sudo iptables -t mangle -A INPUT -s 1.1.1.1 -j LOG --log-prefix="vrf-test-localin" sudo iptables -t mangle -A OUTPUT -s 1.1.1.1 -j LOG --log-prefix="vrf-test-localout" sudo iptables -t mangle -A OUTPUT -d 1.1.1.1 -j LOG --log-prefix="vrf-test-localout"
外网访问本机
ns1 ping 网关1.1.1.254
admin@ubuntu:~$ sudo ip netns exec ns1 ping 1.1.1.254 -c 1 PING 1.1.1.254 (1.1.1.254) 56(84) bytes of data. 64 bytes from 1.1.1.254: icmp_seq=1 ttl=64 time=0.064 ms --- 1.1.1.254 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.064/0.064/0.064/0.000 ms admin@ubuntu:~$
查看log
Nov 20 20:34:10 ubuntu kernel: [180403.527204] vrf-test-preroutingIN=ns1veth1 OUT= MAC=b2:f8:2a:13:31:75:6e:17:d5:b2:55:14:08:00 SRC=1.1.1.1 DST=1.1.1.254 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32492 DF PROTO=ICMP TYPE=8 CODE=0 ID=33955 SEQ=1 Nov 20 20:34:10 ubuntu kernel: [180403.527213] vrf-test-preroutingIN=vrftest OUT= MAC=b2:f8:2a:13:31:75:6e:17:d5:b2:55:14:08:00 SRC=1.1.1.1 DST=1.1.1.254 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32492 DF PROTO=ICMP TYPE=8 CODE=0 ID=33955 SEQ=1 Nov 20 20:34:10 ubuntu kernel: [180403.527220] vrf-test-localinIN=vrftest OUT= MAC=b2:f8:2a:13:31:75:6e:17:d5:b2:55:14:08:00 SRC=1.1.1.1 DST=1.1.1.254 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32492 DF PROTO=ICMP TYPE=8 CODE=0 ID=33955 SEQ=1 Nov 20 20:34:10 ubuntu kernel: [180403.527231] vrf-test-localoutIN= OUT=vrftest SRC=1.1.1.254 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=54845 PROTO=ICMP TYPE=0 CODE=0 ID=33955 SEQ=1 Nov 20 20:34:10 ubuntu kernel: [180403.527233] vrf-test-postroutingIN= OUT=vrftest SRC=1.1.1.254 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=54845 PROTO=ICMP TYPE=0 CODE=0 ID=33955 SEQ=1 Nov 20 20:34:10 ubuntu kernel: [180403.527235] vrf-test-localoutIN= OUT=ns1veth1 SRC=1.1.1.254 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=54845 PROTO=ICMP TYPE=0 CODE=0 ID=33955 SEQ=1 Nov 20 20:34:10 ubuntu kernel: [180403.527242] vrf-test-postroutingIN= OUT=ns1veth1 SRC=1.1.1.254 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=54845 PROTO=ICMP TYPE=0 CODE=0 ID=33955 SEQ=1
从log可以看出：
请求报文经过的hook点顺序如下：
| 序号 | hook点 | 输入接口 | 输出接口 |
|---|---|---|---|
| 1 | PREROUTING | ns1veth1 | 无 |
| 2 | PREROUTING | vrftest | 无 |
| 3 | INPUT | vrftest | 无 |
应答报文经过的hook点顺序如下：
| 序号 | hook点 | 输入接口 | 输出接口 |
|---|---|---|---|
| 1 | OUTPUT | 无 | vrftest |
| 2 | POSTROUTING | 无 | vrftest |
| 3 | OUTPUT | 无 | ns1veth1 |
| 4 | POSTROUTING | 无 | ns1veth1 |
转发报文
ns1 ping ns2
admin@ubuntu:~$ sudo ip netns exec ns1 ping 2.2.2.1 -c 1 PING 2.2.2.1 (2.2.2.1) 56(84) bytes of data. 64 bytes from 2.2.2.1: icmp_seq=1 ttl=63 time=0.063 ms --- 2.2.2.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.063/0.063/0.063/0.000 ms admin@ubuntu:~$
查看log
Nov 20 20:28:31 ubuntu kernel: [180065.076713] vrf-test-preroutingIN=ns1veth1 OUT= MAC=b2:f8:2a:13:31:75:6e:17:d5:b2:55:14:08:00 SRC=1.1.1.1 DST=2.2.2.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=38312 DF PROTO=ICMP TYPE=8 CODE=0 ID=33948 SEQ=1 Nov 20 20:28:31 ubuntu kernel: [180065.076722] vrf-test-preroutingIN=vrftest OUT= MAC=b2:f8:2a:13:31:75:6e:17:d5:b2:55:14:08:00 SRC=1.1.1.1 DST=2.2.2.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=38312 DF PROTO=ICMP TYPE=8 CODE=0 ID=33948 SEQ=1 Nov 20 20:28:31 ubuntu kernel: [180065.076730] vrf-test-forwardIN=vrftest OUT=ns2veth1 MAC=b2:f8:2a:13:31:75:6e:17:d5:b2:55:14:08:00 SRC=1.1.1.1 DST=2.2.2.1 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=38312 DF PROTO=ICMP TYPE=8 CODE=0 ID=33948 SEQ=1 Nov 20 20:28:31 ubuntu kernel: [180065.076732] vrf-test-postroutingIN= OUT=ns2veth1 SRC=1.1.1.1 DST=2.2.2.1 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=38312 DF PROTO=ICMP TYPE=8 CODE=0 ID=33948 SEQ=1 Nov 20 20:28:31 ubuntu kernel: [180065.076746] vrf-test-preroutingIN=ns2veth1 OUT= MAC=02:25:0e:fe:52:35:ba:19:4d:37:ac:8b:08:00 SRC=2.2.2.1 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47601 PROTO=ICMP TYPE=0 CODE=0 ID=33948 SEQ=1 Nov 20 20:28:31 ubuntu kernel: [180065.076749] vrf-test-preroutingIN=vrftest OUT= MAC=02:25:0e:fe:52:35:ba:19:4d:37:ac:8b:08:00 SRC=2.2.2.1 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47601 PROTO=ICMP TYPE=0 CODE=0 ID=33948 SEQ=1 Nov 20 20:28:31 ubuntu kernel: [180065.076752] vrf-test-forwardIN=vrftest OUT=ns1veth1 MAC=02:25:0e:fe:52:35:ba:19:4d:37:ac:8b:08:00 SRC=2.2.2.1 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=47601 PROTO=ICMP TYPE=0 CODE=0 ID=33948 SEQ=1 Nov 20 20:28:31 ubuntu kernel: [180065.076753] vrf-test-postroutingIN= OUT=ns1veth1 SRC=2.2.2.1 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=47601 PROTO=ICMP TYPE=0 CODE=0 ID=33948 SEQ=1
从log可以看出：
请求报文经过的hook点顺序如下：
| 序号 | hook点 | 输入接口 | 输出接口 |
|---|---|---|---|
| 1 | PREROUTING | ns1veth1 | 无 |
| 2 | PREROUTING | vrftest | 无 |
| 3 | FORWARD | vrftest | ns2veth1 |
| 4 | POSTROUTING | 无 | ns2veth1 |
应答报文经过的hook点顺序如下：
| 序号 | hook点 | 输入接口 | 输出接口 | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | PREROUTING | ns2veth1 |
|
||||||||||||||||||||
| 2 | PREROUTING | vrftest | |||||||||||||||||||||
| 3 | FORWARD | vrftest | ns2veth1 | ||||||||||||||||||||
| 4 | POSTROUTING | 无 | ns2veth1 |
vrftest ping 1.1.1.1
admin@ubuntu:~$ sudo ping 1.1.1.1 -I vrftest -c 1 ping: Warning: source address might be selected on device other than vrftest. PING 1.1.1.1 (1.1.1.1) from 1.1.1.254 vrftest: 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=0.039 ms --- 1.1.1.1 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.039/0.039/0.039/0.000 ms admin@ubuntu:~$
查看log
Nov 20 21:21:11 ubuntu kernel: [183224.956734] vrf-test-localoutIN= OUT=vrftest SRC=1.1.1.254 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=35042 DF PROTO=ICMP TYPE=8 CODE=0 ID=34186 SEQ=1 Nov 20 21:21:11 ubuntu kernel: [183224.956740] vrf-test-postroutingIN= OUT=vrftest SRC=1.1.1.254 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=35042 DF PROTO=ICMP TYPE=8 CODE=0 ID=34186 SEQ=1 Nov 20 21:21:11 ubuntu kernel: [183224.956745] vrf-test-localoutIN= OUT=ns1veth1 SRC=1.1.1.254 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=35042 DF PROTO=ICMP TYPE=8 CODE=0 ID=34186 SEQ=1 Nov 20 21:21:11 ubuntu kernel: [183224.956746] vrf-test-postroutingIN= OUT=ns1veth1 SRC=1.1.1.254 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=35042 DF PROTO=ICMP TYPE=8 CODE=0 ID=34186 SEQ=1 Nov 20 21:21:11 ubuntu kernel: [183224.956762] vrf-test-preroutingIN=ns1veth1 OUT= MAC=b2:f8:2a:13:31:75:6e:17:d5:b2:55:14:08:00 SRC=1.1.1.1 DST=1.1.1.254 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=323 PROTO=ICMP TYPE=0 CODE=0 ID=34186 SEQ=1 Nov 20 21:21:11 ubuntu kernel: [183224.956765] vrf-test-preroutingIN=vrftest OUT= MAC=b2:f8:2a:13:31:75:6e:17:d5:b2:55:14:08:00 SRC=1.1.1.1 DST=1.1.1.254 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=323 PROTO=ICMP TYPE=0 CODE=0 ID=34186 SEQ=1 Nov 20 21:21:11 ubuntu kernel: [183224.956769] vrf-test-localinIN=vrftest OUT= MAC=b2:f8:2a:13:31:75:6e:17:d5:b2:55:14:08:00 SRC=1.1.1.1 DST=1.1.1.254 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=323 PROTO=ICMP TYPE=0 CODE=0 ID=34186 SEQ=1从log可以看出：请求报文经过的hook点顺序如下：
| 序号 | hook点 | 输入接口 | 输出接口 |
|---|---|---|---|
| 1 | OUTPUT | 无 | vrftest |
| 2 | POSTROUTING | 无 | vrftest |
| 3 | OUTPUT | 无 | ns1veth1 |
| 4 | POSTROUTING | 无 | ns1veth1 |
应答报文经过的hook点顺序如下：
| 序号 | hook点 | 输入接口 | 输出接口 |
|---|---|---|---|
| 1 | PREROUTING | ns1veth1 | 无 |
| 2 | PREROUTING | vrftest | 无 |
| 3 | INPUT | vrftest | 无 |
本机访问本机，即环回
vrftest ping vrf接口地址9.9.9.9/24
添加如下netfilter规则：
admin@ubuntu:~/vrftcpdump$ cat netfilter1.sh sudo iptables -t mangle -A PREROUTING -s 9.9.9.9 -j LOG --log-prefix="vrf-test-prerouting" sudo iptables -t mangle -A FORWARD -s 9.9.9.9 -j LOG --log-prefix="vrf-test-forward" sudo iptables -t mangle -A POSTROUTING -s 9.9.9.9 -j LOG --log-prefix="vrf-test-postrouting" sudo iptables -t mangle -A INPUT -s 9.9.9.9 -j LOG --log-prefix="vrf-test-localin" sudo iptables -t mangle -A OUTPUT -s 9.9.9.9 -j LOG --log-prefix="vrf-test-localout" admin@ubuntu:~/vrftcpdump$
配置vrftest接口地址为9.9.9.9/24
admin@ubuntu:~/vrftcpdump$ sudo ip addr add 9.9.9.9/24 dev vrftest admin@ubuntu:~/vrftcpdump$ sudo ping 9.9.9.9 -I vrftest -c 1 PING 9.9.9.9 (9.9.9.9) from 9.9.9.9 vrftest: 56(84) bytes of data. 64 bytes from 9.9.9.9: icmp_seq=1 ttl=64 time=0.050 ms --- 9.9.9.9 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.050/0.050/0.050/0.000 ms admin@ubuntu:~/vrftcpdump$
查看log
Nov 20 22:13:41 ubuntu kernel: [186374.589186] vrf-test-localoutIN= OUT=vrftest SRC=9.9.9.9 DST=9.9.9.9 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=39425 DF PROTO=ICMP TYPE=8 CODE=0 ID=34815 SEQ=1 Nov 20 22:13:41 ubuntu kernel: [186374.589192] vrf-test-postroutingIN= OUT=vrftest SRC=9.9.9.9 DST=9.9.9.9 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=39425 DF PROTO=ICMP TYPE=8 CODE=0 ID=34815 SEQ=1 Nov 20 22:13:41 ubuntu kernel: [186374.589202] vrf-test-preroutingIN=vrftest OUT= MAC=ca:f9:f0:37:4c:6c:ca:f9:f0:37:4c:6c:08:00 SRC=9.9.9.9 DST=9.9.9.9 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=39425 DF PROTO=ICMP TYPE=8 CODE=0 ID=34815 SEQ=1 Nov 20 22:13:41 ubuntu kernel: [186374.589204] vrf-test-localinIN=vrftest OUT= MAC=ca:f9:f0:37:4c:6c:ca:f9:f0:37:4c:6c:08:00 SRC=9.9.9.9 DST=9.9.9.9 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=39425 DF PROTO=ICMP TYPE=8 CODE=0 ID=34815 SEQ=1 Nov 20 22:13:41 ubuntu kernel: [186374.589210] vrf-test-localoutIN= OUT=vrftest SRC=9.9.9.9 DST=9.9.9.9 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=39426 PROTO=ICMP TYPE=0 CODE=0 ID=34815 SEQ=1 Nov 20 22:13:41 ubuntu kernel: [186374.589211] vrf-test-postroutingIN= OUT=vrftest SRC=9.9.9.9 DST=9.9.9.9 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=39426 PROTO=ICMP TYPE=0 CODE=0 ID=34815 SEQ=1 Nov 20 22:13:41 ubuntu kernel: [186374.589215] vrf-test-preroutingIN=vrftest OUT= MAC=ca:f9:f0:37:4c:6c:ca:f9:f0:37:4c:6c:08:00 SRC=9.9.9.9 DST=9.9.9.9 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=39426 PROTO=ICMP TYPE=0 CODE=0 ID=34815 SEQ=1 Nov 20 22:13:41 ubuntu kernel: [186374.589217] vrf-test-localinIN=vrftest OUT= MAC=ca:f9:f0:37:4c:6c:ca:f9:f0:37:4c:6c:08:00 SRC=9.9.9.9 DST=9.9.9.9 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=39426 PROTO=ICMP TYPE=0 CODE=0 ID=34815 SEQ=1
从log可以看出：
请求报文经过的hook点顺序如下：
| 序号 | hook点 | 输入接口 | 输出接口 |
|---|---|---|---|
| 1 | OUTPUT | 无 | vrftest |
| 2 | POSTROUTING | 无 | vrftest |
| 3 | PREROUTING | vrftest | 无 |
| 4 | INPUT | vrftest | 无 |
应答报文经过的hook点顺序如下：
| 序号 | hook点 | 输入接口 | 输出接口 |
|---|---|---|---|
| 1 | OUTPUT | 无 | vrftest |
| 2 | POSTROUTING | 无 | vrftest |
| 3 | PREROUTING | vrftest | 无 |
| 4 | INPUT | vrftest | 无 |
环回请求和应答是一样的。
总结
- linux内核支持VRF后，报文会两次进入PREROUTING节点，第一次是原始接口，第二次是原始接口依附的vrf主接口。
- linux内核支持VRF后，本机输出的报文，先以vrf接口作为输出接口经过OUTPUT和POSTROUTING节点，然后再以真实出去的接口经过OUPUT和POSTROUTING节点。
- 本机报文进行环回时，与没有VRF的环境是一样的。
