Update: Gentoo Routing Issue

[dhedge]

Update: Gentoo Routing Issue

Here is a recap of the problem:
I have two identically configured production web servers that are working just fine each with two network interfaces. Interface A connected to the 192.168.0 subnet on the private network. Interface B connected the public network, the default gateway, a central router all our networks are connected to, is on the public network. Either interface is reachable from any machine on our private and public network. Took one down and updated it with emerge. Brought it back up and now machines on the 192.168.0 subnet can only access the machine through interface A, and machines on the public network as well as the 192.168.2 subnet, the 192.168.0 and 192.168.2 subnets comprise the private network, on the private network can only access the machine through interface B. If you bring down interface A, the 192.168.0 private subnet, then the 192.168.0, 192.168.2, and the public networks can all reach the machine through interface B no problems.

P.S. Previously we had seen other odd behaviour. Sometimes on a restart you couldn't communicate over either of the network interfaces at all, until you change the IP address, just the ipaddress, then it worked just fine. Reassign the old IP Address and it again becomes inaccessible. Give our other non-updated web server the same IP's and it can communicate just fine. Tried powercycling the switiches, still no effect.

Here are the results of our latest tests:
1. Swaped in a new harddrive and did a fresh stage 3 install of gentoo. Configured both interfaces and the same behaviour is showing. Nothing besides the base gentoo install is on the system.

2. Wiped the drive and installed Solaris 10 x86. Configured both interfaces and problem solved. Both interfaces are accessible from all networks at the same time.

Questions:
Below is a hardware spec list of this system. Has anyone else with the same chipsets seen this? Has anyone else seen this behaviour with gentoo? Were you able to isolate the exact cause? Were you able to resolve the issue?

At the moment it looks to us like it is most likely a driver issue.

Hardware Specs

Motherboard:
SuperMicro X5DLR-8G2
Serverworks™ GC-LE Chipset
Adaptec AIC-7902 Controller
• Dual-Channel Ultra320 SCSI
Dual Broadcom 5704 Gigabit Ethernet
ATI Rage XL SVGA PCI video controller with 8 MB of video memory

Processor: Intel Pentium 4 Xeon

PLEASE LOOK AT PREVIOUS POST FOR DETAILS ON NETWORK CONFIGURATION:
http://www.linuxforums.org/forum/lin...tml#post294841

[vrejakti]

I have had this exact same issue with Gentoo Linux, and Gentoo only. The funky stuff occurs when you install a second Ethernet interface. I haven't isolated the issue, but I can provide very useful information:

****My Configuration:

`ifconfig` info on eth0:
inet addr:172.16.0.2
Bcast:172.16.0.3
Mask:255.255.255.252

`ifconfig` info on eth1:

inet addr:192.168.0.150
Bcast:192.168.0.255
Mask:255.255.255.0

****Settings from /etc/conf.d/net

###################################
# 172.16.0.1 = Smoothwall Orange #

config_eth0=( "172.16.0.2 netmask 255.255.255.252 brd 172.16.0.3" )
routes_eth0=( "default via 172.16.0.1" )

################################
# Use Routes for SSH only temp #

config_eth1=( "192.168.0.150/24" )
##routes_eth1=( "default gw 192.168.0.1" )

****Settings from rc-status

Runlevel: default
sshd [ started ]
local [ started ]
mysql [ started ]
syslog-ng [ started ]
coldplug [ started ]
netmount [ started ]
apache2 [ started ]

So, what I have found...

Problem: if you have two interfaces configured to load on run level default, you will have conflicting gateways, and will be unable to route traffic anywhere useful.
Solution: Remove net.eth#'s from boot. Apache or SSH will load the interfaces for you (since they are required).

Problem: Only one interface is reachable from the outside world.
Explination: That interface would be the default highest priority gateway.

Some tests using hping:
TEST1
kane vrejakti # hping2 192.168.0.111
HPING 192.168.0.111 (eth1 192.168.0.111): NO FLAGS are set, 40 headers + 0 data bytes
ICMP Host Unreachable from ip=192.168.0.150 get hostname...

TEST2
HPING google.com (eth0 64.233.167.99): NO FLAGS are set, 40 headers + 0 data bytes
ICMP Port Unreachable from ip=172.16.0.1 name=UNKNOWN

TEST3
kane vrejakti # hping2 192.168.1.191
HPING 192.168.1.191 (eth0 192.168.1.191): NO FLAGS are set, 40 headers + 0 data bytes
ICMP Port Unreachable from ip=172.16.0.1 name=UNKNOWN

Notice how on tests 1, the IP pinged is using eth1, the interface from that IP. Now as soon as you enter an IP with a different subnet it immetialy switches to the default gateway of eth0 as shown in tests 2 and 3.

Other tests of interest, attempting to SSH into my Smoothwall Firewall's green zone that's behind eth1 I am blocked because the IP isn't on 192.168.0.xxx. However HTTP-ing into the router address of eth1 being 192.168.0.1 access is granted.

kane vrejakti # telnet 10.10.133.7 222
Trying 10.10.133.7...
telnet: connect to address 10.10.133.7: Connection refused

kane vrejakti # telnet 192.168.0.1 80
Trying 192.168.0.1...
Connected to 192.168.0.1.


All this info aside, I feel the solution would be to extend the address pool that the secondary interfaces route. It seems by default the secondary interface will only route addresses 192.168.0.0 - 192.168.0.255 thought the interface sending all other requests through the default gateway. How to make such an extension, I have no clue. ^_^ Prob find the answer in the advanced networking section of the Gentoo Handbook.

[dhedge]

We've done some more testing and considerably narrowed the possibilites as well as showing a larger issue than was previously thought. The observed connection issues show up consistently on multiple distributions of Linux and on widely separate hardware.

Older Linux distributions do not show this bug so it is most likely a recent update that is causing the behaviour.

We tried all distributions listed on both machines listed. They both plugged into the same network cables and ports on the same switches for all tests. When they worked, clients on all subnets were able to access both interfaces. When they didn't work, clients were only able to access the port that they shared a subnet with. Details are in our previous posts.

This is now obviously not a hardware or distribution specific problem so I'm not sure where to go for help. Does anyone here have either a suggestion for how to fix the problem or a suggestion of where to post a bug report or help request that might be more appropriate?

Linux:
Gentoo 2005.0 - Works
Gentoo 2005.1 - Does not work
KNOPPIX V4.0.2 2005-09-23 - Works
RHEL4-U3-i386-ES - Works
Slackware 10.2 - Does not work
Ubuntu 5.10 - Does not work

Unix:
FreeBSD 5.3 - Works
Solaris 10 - Works

Machine 1:
SuperMicro X5DLR-8G2
Serverworks GC-LE Chipset
Adaptec AIC-7902 Controller
Dual-Channel Ultra320 SCSI
Dual Broadcom 5704 Gigabit Ethernet
ATI Rage XL SVGA PCI video controller with 8 MB of video memory
Dual Intel Pentium 4 Xeon Processors
(Also tried Netgeat FA-311TX & Dual-Port Intel Network cards in this machine with the same results)

Machine 2:
Matsonic MS7308ET
SiS630ET Chipset (onboard Nic)
3com 3C905B-TX NIC
Single Intel Pentium 3 Processor