Multiple interface on same network using same NIC for communication

[tushs]

I am facing interesting problem, My setup is as follows,
Machine-1
- Two network interface cards on same network with different MAC say MAC-1 and MAC-2
- IP 10.209.193.130 and 10.209.193.131
- Firewall IP tables empty iptables -F
Machine-2
- One NIC with IP 10.209.193.135
switch
- both machines are connected to common switch and can ping happily.

Problem:
when I send ping request from Machine-2 to Machine-1 arp tables are initialized for IP MAC pair. Surprisingly both IP are get resolved to one MAC i.e MAC-1. general observation is, MAC of first interface which has route in kernel route table.
I cross checked this on Suse11 and RHEL5 same result. but In suse10 I am getting expected behavior [i.e IP1-MAC1 and IP2-MAC2 pair in arp cache].
Do any one know why kernel is using same NIC for two different interfaces?

Output for Suse11 and RHEL 5

? (10.209.193.131) at 00:0c:29:34:e7:6f [ether] on eth3
? (10.209.193.130) at 00:0c:29:34:e7:6f [ether] on eth3

Output for Suse10

? (10.209.193.128) at 00:0c:29:02:9a:7c [ether] on eth3   <-------- Suse 10 machine-1
? (10.209.193.131) at 00:0c:29:34:e7:6f [ether] on eth3
? (10.209.193.127) at 00:0c:29:02:9a:72 [ether] on eth3 <-------- Suse 10 machine-2
? (10.209.192.1) at 00:00:0c:07:ac:07 [ether] on eth3

[Ziplock]

Hi tushs,

It could be a couple of things, since the ips are on the same subnet.

I would check that the interfaces are specifically assigned to the Mac addresses they are sitting on. Use the HWADDR variable to assign the Mac address in /etc/sysconfig/network-scripts/ifcfg-eth0 & eth1

I think the kernel gets "smart" with ips on the same network.

Can you still ping both IPs if you unplug eth1?

[tushs]

Let me tell u how I got mess up with this,
-both eth0 and eth1 are working on same subnet
-both are ping-able
-unplug cable of eth0
-IP on eth0 is not ping able but IP on eth1 is reachable
-No replug eth0 cable
---- what ever I do IP on eth0 is not reachable
I tried to update arp cache of neighbors using
arping -A -c 3 <IP of eth0> -I eth0
It updates cache but still not ping able. after ping command arp cache is "incomplete"
I am not getting why kernel is not responding to "who has <IP> " request after unplug - replug cable !!!!
Does any one know this?

[HROAdmin26]

Do not put 2 NIC's on the same switch/subnet and expect good results. This leads to a loop condition and inconsistent behavior. If you want 2 NIC's for failover/throughput, do it correctly and BOND them together.

[tushs]

These two interfaces are getting IP by DHCP server, How can I set these interface on different network using DHCP In case of static IP it is possible. also this might be possible that I can use 3rd interface then in that case I need yet new network or subnet!!
there is no way to keep things on same subnet with high availability and throughput ?? can I do some thing using iptables?

[Irithori]

As HROAdmin26 said, have a look at nic bonding.
This is the redhat 6 documentation:
4.2.2.*Channel Bonding Interfaces

[Ziplock]

I agree, I didn't realize you were trying to do failover, bonding is definitely the way to go. Depending on the brand of switch you have there may be some configuration on that as well.