IPSec + GRE/openvswitch equals poor performance

[maciejgalkiewicz]

I have two machines with Debian squeeze (kernel 2.6.32-5-amd64) connected via GRE tunnel on top of IPSec transport mode (openswan) with 3des encryption.

|m1|-eth0------------------|cloud|------------------eth0-|m2|
   |                                                      |
   |--gre0-----------------IPSec + GRE---------------gre0-|
                         192.168.1.0/24

Both eth0 interfaces are 100Mbit Ethernet.

I have made some test with scp and iperf tools:
1. Connection without IPSec via eth0

#time scp file m2:
file                 100%  271MB  11.3MB/s   00:24    

real    0m27.425s
user    0m3.052s
sys     0m0.392s

# iperf -c m2 -p 6666
------------------------------------------------------------
Client connecting to m2, TCP port 6666
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local m1 port 51003 connected with m2 port 6666
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec    112 MBytes  94.0 Mbits/sec

2. Connection without IPSec via GRE tunnel

#time scp file m2:
file                 100%  271MB  10.1MB/s   00:27    

real    0m34.369s
user    0m3.032s
sys     0m0.460s

# iperf -c m2 -p 6666
------------------------------------------------------------
Client connecting to m2, TCP port 6666
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local m1 port 51003 connected with m2 port 6666
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec    109 MBytes  91.1 Mbits/sec

3. Connection with IPSec via eth0

#time scp file m2:
file                 100%  271MB  10.9MB/s   00:25    

real    0m28.075s
user    0m3.064s
sys     0m1.952s

# iperf -c m2 -p 6666
------------------------------------------------------------
Client connecting to m2, TCP port 6666
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local m1 port 51003 connected with m2 port 6666
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec    109 MBytes  91.6 Mbits/sec

4. Connection with IPSec via GRE tunnel

#time scp file m2:
file                 9%   27MB 452.4KB/s   09:13 ETA^C

real    1m1.899s
user    0m0.352s
sys     0m56.400s

# iperf -c m2 -p 6666 -i 10 -t 60
------------------------------------------------------------
Client connecting to m2, TCP port 6666
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  3] local m1 port 37725 connected with m2 port 6666
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  10.4 MBytes  8.71 Mbits/sec
[  3] 10.0-20.0 sec  8.54 MBytes  7.16 Mbits/sec
[  3] 20.0-30.0 sec  7.47 MBytes  6.27 Mbits/sec
[  3] 30.0-40.0 sec  6.75 MBytes  5.66 Mbits/sec
[  3] 40.0-50.0 sec  6.20 MBytes  5.20 Mbits/sec
[  3] 50.0-60.0 sec  5.75 MBytes  4.82 Mbits/sec
[  3]  0.0-60.0 sec  45.1 MBytes  6.30 Mbits/sec

What is wrong with 4th test? Why the connection is so slow? Both tools uses 100% cpu time during only this trial. My servers have i7-920 (2.67GHz). I have also tried with openvswitch software which is able to send Ethernet over GRE. The results were the same. The last idea which came to my mind was openvpn. I set up openvpn in bridge mode also on top of IPSec. The results were only a little worse than the third trial. Any suggestions? Maybe it is a bug in linux kernel?

best regards
Maciej Galkiewicz

[Kloschüssel]

Looks to me as if IPSec over GRE is being done in software and not in hardware (IPSec over eth0 probably is done in hardware). Encryption can be really slow if it is not hardware accelerated.

EDIT:

in fact my assumptions seem to be right:

A quote from NIL - Data Center Interconnections: Technical Implementations yields following:

GRE over IPSec over the Internet
[...]
To achieve high performance, this technology requires special hardware, such as VPN acceleration modules to encrypt and decrypt traffic at near-wire speeds.

[maciejgalkiewicz]

Looks to me as if IPSec over GRE is being done in software and not in hardware (IPSec over eth0 probably is done in hardware). Encryption can be really slow if it is not hardware accelerated.

This is GRE over IPSec not conversely. I don't think that there is hardware acceleration. My server has RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 02). BTW the cpu i7-920 is really efficient.

[Kloschüssel]

This is GRE over IPSec not conversely.

My error, sorry.

My server has RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 02). BTW the cpu i7-920 is really efficient.

Maybe packet fragmentation is the evil. Try to decrease the MTU size in your tunnel by 68 bytes.