GRE tunnel troubleshooting

[solarflare]

I am trying to setup GRE between two CentOS 4.5 boxes. I have tried several variations of what's listed below, but none of them work.

box1:
modprobe ip_gre
ip link set gre0 up
ip tunnel add gretun mode gre local 66.1.1.161 remote 66.1.2.161 ttl 20 dev eth0
ip addr add dev gretun 10.253.253.1 peer 10.253.253.2/24
ip link set dev gretun up
ip route add 10.2.0.0/16 via 10.253.253.2

box2:
modprobe ip_gre
ip link set gre0 up
ip tunnel add gretun mode gre local 66.1.2.161 remote 66.1.1.161 ttl 20 dev eth0
ip addr add dev gretun 10.253.253.2 peer 10.253.253.1/24
ip link set dev gretun up
ip route add 10.1.0.0/16 via 10.253.253.1

tcpdump shows NO rx or tx traffic from either box that isn't ARP or SSH.

It's like it's not even trying to bring the tunnel up. I'm a Cisco guy, so I'm lost with my show commands.

The other variations I've tried consist mostly of trying different combinations of on-net (in the same subnet as eth0 and even the same address as eth0) and off-net (various combinations of loopback /24 and /32 addresses in separate 10 space) on the 'ip addr add dev gretun' statements. But the above example is what *should* work on a Cisco, I think. It's been a while.

How do I troubleshoot this?

root@den1tun01:/home/root $ ip link
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP> mtu 8800 qdisc pfifo_fast qlen 1000
link/ether 00:19:b9:dd:ff:d9 brd ff:ff:ff:ff:ff:ff
3: eth0.2: <BROADCAST,MULTICAST,UP> mtu 8800 qdisc noqueue
link/ether 00:19:b9:dd:ff:d9 brd ff:ff:ff:ff:ff:ff
4: gre0: <NOARP,UP> mtu 1476 qdisc noqueue
link/gre 0.0.0.0 brd 0.0.0.0
5: gretun@eth0: <POINTOPOINT,NOARP,UP> mtu 8776 qdisc noqueue
link/gre 66.1.1.161 peer 66.1.2.161

root@den1tun01:/home/root $ ip tun
gre0: gre/ip remote any local any ttl inherit nopmtudisc
gretun: gre/ip remote 66.1.2.161 local 66.1.1.161 dev eth0 ttl 20

root@den1tun01:/home/root $ ifconfig
eth0 Link encap:Ethernet HWaddr 00:19:B9D:FF9
inet addr:10.1.2.243 Bcast:10.1.3.255 Mask:255.255.254.0
UP BROADCAST RUNNING MULTICAST MTU:8800 Metric:1
RX packets:3357 errors:0 dropped:0 overruns:0 frame:0
TX packets:484 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:230757 (225.3 KiB) TX bytes:63937 (62.4 KiB)
Interrupt:169 Memory:f8000000-f8011100

eth0.2 Link encap:Ethernet HWaddr 00:19:B9D:FF9
inet addr:66.1.1.161 Bcast:66.1.1.191 Mask:255.255.255.192
UP BROADCAST RUNNING MULTICAST MTU:8800 Metric:1
RX packets:950 errors:0 dropped:0 overruns:0 frame:0
TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:43860 (42.8 KiB) TX bytes:1200 (1.1 KiB)

gretun Link encap:UNSPEC HWaddr 42-0B-33-A1-FF-C0-00-00-00-00-00-00-00-00-00-00
inet addr:10.253.253.1 P-t-P:10.253.253.2 Mask:255.255.255.0
UP POINTOPOINT RUNNING NOARP MTU:8776 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:756 (756.0 b)

gre0 Link encap:UNSPEC HWaddr 00-00-00-00-FF-00-00-00-00-00-00-00-00-00-00-00
UP RUNNING NOARP MTU:1476 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:225 errors:0 dropped:0 overruns:0 frame:0
TX packets:225 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:13271 (12.9 KiB) TX bytes:13271 (12.9 KiB)

Thanks!
Greg

[solarflare]

I've also tried changing the destination for the route to the near end of the private subnet and tried pinging various things on the tunnel subnet and remote network to create "interesting traffic" to bring the tunnel up but tcpdump shows nothing.

Then I noticed that ping does show an error count:

[root@den1tun01 ~]# ping 10.253.253.2
PING 10.253.253.2 (10.253.253.2) 56(84) bytes of data.
From 10.253.253.1 icmp_seq=0 Destination Host Unreachable
From 10.253.253.1 icmp_seq=1 Destination Host Unreachable

--- 10.253.253.2 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1000ms
, pipe 2

I can ping the local end: 10.253.253.1

[solarflare]

Finally, a hint of light:

The first is a tcpdump while pinging the remote end, 66.1.2.161

10:12:10.441842 > 00:19:b9:dd:ff:d9 ip 100: IP 66.1.1.161 > 66.1.2.161: icmp 64: echo request seq 1
10:12:10.442344 < 00:01:e8:0f:ee:f8 ip 100: IP 66.1.2.161 > 66.1.1.161: icmp 64: echo reply seq 1

This next is a ping of the remote tunnel end, 10.253.253.2

10:12:18.970786 > 00:19:b9:dd:ff:d9 arp 44: arp who-has 66.1.2.161 tell 66.1.1.161

I am *very* confused by this. Somehow, when I try to send traffic thru the tunnel, it thinks that the remote physical end is directly attached and should ARP for it even tho it is pingable?!?!!? It is definitely not on-net - it is many hops away - but it is reachable via a default route.

Routing table before the tunnel is configured:

[root@den1tun01 ~]# netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
66.1.1.128 0.0.0.0 255.255.255.192 U 0 0 0 eth0.2
10.1.2.0 0.0.0.0 255.255.254.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0.2
10.0.0.0 10.1.2.254 255.0.0.0 UG 0 0 0 eth0
0.0.0.0 66.1.1.129 0.0.0.0 UG 0 0 0 eth0.2
[root@den1tun01 ~]#

And while it's configured:

[root@den1tun01 ~]# netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
66.1.1.128 0.0.0.0 255.255.255.192 U 0 0 0 eth0.2
10.253.253.0 0.0.0.0 255.255.255.0 U 0 0 0 gretun
10.1.2.0 0.0.0.0 255.255.254.0 U 0 0 0 eth0
10.50.0.0 0.0.0.0 255.255.0.0 U 0 0 0 gretun
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0.2
10.0.0.0 10.1.2.254 255.0.0.0 UG 0 0 0 eth0
0.0.0.0 66.1.1.129 0.0.0.0 UG 0 0 0 eth0.2

[solarflare]

The problem appears to be a bug in the kernel. One thing I'd forgotten to mention was that I regularly use VLAN interfaces on all of my machines. The public exit interface on the CentOS boxes was a VLAN tagged interface (eth0.2). But when I did later tests on the Fedora boxes, I just did a test on their native interfaces, which worked. At first I thought perhaps I was running into an MTU problem, but then I realized I was only sending small ICMP packets to test. And tcpdump isn't seeing any packets exit so I *think* Linux is choking on the double tagging.

If I exchange the addresses, moving the public address to the untagged interface (eth0) and the private address to the tagged interface (eth0.2 for example), GRE works fine since the default route is using the publicly addressed interface and is much happier when it's untagged.

Is this a bug or are 802.1q and GRE tagging just inherently incompatible?

Greg