Hi, i have an extremely strange problem and i am completely stumped.

My ISP has assigned me the ip address 83.111.160.6 (on a /30 subnet, gw is 83.111.160.5), and they route an additional block 83.111.196.56/29 (which gives me 83.111.196.57 - 83.111.196.62 useable addresses).

I have a debian linux box, with the ip address 83.111.160.6 connecting to a media converter on an ethernet port. The routed block are setup as address aliases on the interface. I am running Shorewall and have setup the box to accept ssh and ping for each ip alias. This works and i am 99.999% sure this isnt a shorewall issue at all for reasons i will explain below.
Code:
/etc/network/interfaces
auto eth3
iface eth3 inet static
    address 83.111.160.6
    netmask 255.255.255.252
    up ip addr add 83.111.196.57/29 brd 83.111.196.63 dev eth3 label eth3:0
    up ip addr add 83.111.196.58/29 brd 83.111.196.63 dev eth3 label eth3:1
    up ip addr add 83.111.196.59/29 brd 83.111.196.63 dev eth3 label eth3:2
    up ip addr add 83.111.196.60/29 brd 83.111.196.63 dev eth3 label eth3:3
    up ip addr add 83.111.196.61/29 brd 83.111.196.63 dev eth3 label eth3:4
    up ip addr add 83.111.196.62/29 brd 83.111.196.63 dev eth3 label eth3:5
And here is a snippet from the shorewall rules config (but to restate this is not the issue)
Code:
Ping/ACCEPT     net             $FW
Ping/ACCEPT     net             $FW:83.111.196.57
Ping/ACCEPT     net             $FW:83.111.196.58
Ping/ACCEPT     net             $FW:83.111.196.59
Ping/ACCEPT     net             $FW:83.111.196.60
Ping/ACCEPT     net             $FW:83.111.196.61
Ping/ACCEPT     net             $FW:83.111.196.62

ACCEPT          net             $FW:83.111.160.6        tcp     ssh
ACCEPT         net             $FW:83.111.196.57       tcp     ssh
ACCEPT         net             $FW:83.111.196.58       tcp     ssh
ACCEPT         net             $FW:83.111.196.59       tcp     ssh
ACCEPT         net             $FW:83.111.196.61       tcp     ssh
ACCEPT         net             $FW:83.111.196.62       tcp     ssh
I can ping 83.111.160.6 fine, but i cant ping all of the routed ip addresses. Some work and some dont! With shorewall set to reject icmp and ssh, some of the connection attempts are listed as being dropped, but other addresses are not (namely the ones that were pingable when shorewall was set to allow icmp).

Furthermore, Siteuptime.com shows some sites able to connect to the ip address i assigned as an smtp server (83.111.196.60), and others not able to connect (US sites ok, London failed).

Using network-tools.com i got a number of traceroutes:

Code:
Ping 83.111.196.59
Timed out
Timed out
Timed out

TraceRoute to 83.111.196.59
Hop (ms) (ms) (ms)  IP Address Host name 
1 9 16 18  72.249.0.65 - 
2 35 25 20  64.129.174.181 64-129-174-181.static.twtelecom.net 
3 51 47 45  66.192.242.253 - 
4 Timed out Timed out Timed out   - 
5 280 270 268  195.229.1.186 - 
6 263 265 276  194.170.0.154 - 
7 Timed out Timed out Timed out   - 
8 295 318 301  83.111.206.182 - 
9 Timed out Timed out Timed out   - 
10 Timed out Timed out Timed out   - 

Ping 83.111.196.60
Round trip time to 83.111.196.60: 272 ms
Round trip time to 83.111.196.60: 267 ms
Round trip time to 83.111.196.60: 270 ms
Round trip time to 83.111.196.60: 274 ms
Round trip time to 83.111.196.60: 263 ms
Round trip time to 83.111.196.60: 271 ms
Round trip time to 83.111.196.60: 273 ms
Round trip time to 83.111.196.60: 272 ms
Round trip time to 83.111.196.60: 263 ms
Round trip time to 83.111.196.60: 267 ms
Average time over 10 pings: 269.2 ms

TraceRoute to 83.111.196.60
Hop (ms) (ms) (ms)  IP Address Host name 
1 12 11 5  72.249.0.65 - 
2 11 24 23  64.129.174.181 64-129-174-181.static.twtelecom.net 
3 49 46 59  66.192.242.253 - 
4 Timed out Timed out Timed out   - 
5 275 270 272  195.229.1.186 - 
6 264 263 263  194.170.0.158 - 
7 Timed out Timed out Timed out   - 
8 274 264 269  83.111.196.60 - 

Ping 83.111.196.61
Round trip time to 83.111.196.61: 277 ms
Round trip time to 83.111.196.61: 286 ms
Round trip time to 83.111.196.61: 279 ms
Round trip time to 83.111.196.61: 285 ms
Round trip time to 83.111.196.61: 269 ms
Round trip time to 83.111.196.61: 265 ms
Timed out
Round trip time to 83.111.196.61: 268 ms
Round trip time to 83.111.196.61: 263 ms
Round trip time to 83.111.196.61: 275 ms
Average time over 10 pings: 246.7 ms

TraceRoute to 83.111.196.61
Hop (ms) (ms) (ms)  IP Address Host name 
1 11 19 22  72.249.0.65 - 
2 15 22 16  64.129.174.181 64-129-174-181.static.twtelecom.net 
3 60 50 46  66.192.242.253 - 
4 Timed out Timed out Timed out   - 
5 273 284 282  195.229.1.186 - 
6 264 263 268  194.170.0.158 - 
7 Timed out Timed out Timed out   - 
8 274 272 271  83.111.196.61 - 

Ping 83.111.196.62
Timed out
Timed out
Timed out

TraceRoute to 83.111.196.62
Hop (ms) (ms) (ms)  IP Address Host name 
1 7 5 5  72.249.0.65 - 
2 7 7 12  64.129.174.181 64-129-174-181.static.twtelecom.net 
3 62 51 51  66.192.242.253 - 
4 Timed out Timed out Timed out   - 
5 292 272 275  195.229.1.186 - 
6 271 272 262  194.170.0.158 - 
7 Timed out Timed out Timed out   - 
8 291 281 279  83.111.206.182 - 
9 Timed out Timed out Timed out   - 
10 Timed out Timed out Timed out   - 
11 Timed out Timed out Timed out   - 
12 Timed out Timed out Timed out   -

And using the following tool (traceroute from 193.62.127.224 (icfamon.dl.ac.uk) to your web browser at 67.15.52.42 (ev1s-67-15-52-42.theplanet.com) for 67.15.52.42) the traceroute seemed to timeout:
Code:
traceroute from 193.62.127.224 (icfamon.dl.ac.uk) to 83.111.196.60 

traceroute to 83.111.196.60 (83.111.196.60), 30 hops max, 38 byte packets
 1  alan5 (193.62.127.129)  3.099 ms  0.518 ms  0.475 ms
 2  gw-fw (193.63.74.131)  0.247 ms  0.210 ms  0.215 ms
 3  c-pop (193.63.74.226)  1.033 ms  2.677 ms  0.733 ms
 4  193.62.116.18 (193.62.116.18)  1.229 ms  1.119 ms  1.115 ms
 5  so-0-1-0.warr-sbr1.ja.net (146.97.42.169)  1.720 ms  1.750 ms  1.736 ms
 6  so-3-0-0.lond-sbr3.ja.net (146.97.33.18)  6.720 ms  6.716 ms  6.720 ms
 7  195.219.100.17 (195.219.100.17)  7.121 ms  6.825 ms  6.806 ms
 8  if-13-0-0-3.mcore3.LDN-London.teleglobe.net (195.219.195.21)  25.400 ms  205.769 ms  8.021 ms
 9  Vlan62.icore1.LDN-London.teleglobe.net (195.219.83.1)  15.335 ms  17.154 ms  18.018 ms
10  linx.lon.seabone.net (195.66.224.153)  7.699 ms  8.043 ms  7.779 ms
11  customer-side-etisalat-4-pal9.pal.seabone.net (213.144.181.170)  219.954 ms  217.635 ms  220.075 ms
12  195.229.1.194 (195.229.1.194)  235.358 ms  230.137 ms  230.371 ms
13  194.170.0.158 (194.170.0.158)  293.654 ms  300.541 ms 194.170.0.154 (194.170.0.154)  222.685 ms
14  195.229.245.142 (195.229.245.142)  229.187 ms  221.517 ms  221.703 ms
15  83.111.206.182 (83.111.206.182)  311.977 ms  309.222 ms  311.551 ms
Now here is the strangest thing, i have a couple of servers in the UK and they have dual interfaces. On one of the boxes, ping fails from one interface, but works on another... to the same destination host!! Please see below:
Code:
[root@stripe ~]# ping 83.111.196.59 -I 85.234.115.64
PING 83.111.196.59 (83.111.196.59) from 85.234.115.64 : 56(84) bytes of data.

--- 83.111.196.59 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3002ms

[root@stripe ~]# ping 83.111.196.60 -I 85.234.115.64
PING 83.111.196.60 (83.111.196.60) from 85.234.115.64 : 56(84) bytes of data.
64 bytes from 83.111.196.60: icmp_seq=1 ttl=56 time=159 ms
64 bytes from 83.111.196.60: icmp_seq=2 ttl=56 time=159 ms
--- 83.111.196.60 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 159.024/159.221/159.418/0.197 ms

[root@stripe ~]# ping 83.111.196.61 -I 85.234.115.64
PING 83.111.196.61 (83.111.196.61) from 85.234.115.64 : 56(84) bytes of data.
64 bytes from 83.111.196.61: icmp_seq=1 ttl=54 time=148 ms
64 bytes from 83.111.196.61: icmp_seq=2 ttl=54 time=148 ms
--- 83.111.196.61 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 148.549/148.615/148.681/0.066 ms

[root@stripe ~]# ping 83.111.196.62 -I 85.234.115.64
PING 83.111.196.62 (83.111.196.62) from 85.234.115.64 : 56(84) bytes of data.
--- 83.111.196.62 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2000ms

[root@stripe ~]# ping 83.111.196.59 -I 85.234.115.115
PING 83.111.196.59 (83.111.196.59) from 85.234.115.115 : 56(84) bytes of data.
64 bytes from 83.111.196.59: icmp_seq=1 ttl=57 time=149 ms
64 bytes from 83.111.196.59: icmp_seq=2 ttl=57 time=158 ms
--- 83.111.196.59 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 149.200/153.985/158.771/4.801 ms

[root@stripe ~]# ping 83.111.196.60 -I 85.234.115.115
PING 83.111.196.60 (83.111.196.60) from 85.234.115.115 : 56(84) bytes of data.
--- 83.111.196.60 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 2999ms

[root@stripe ~]# ping 83.111.196.61 -I 85.234.115.115
PING 83.111.196.61 (83.111.196.61) from 85.234.115.115 : 56(84) bytes of data.
--- 83.111.196.61 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2000ms

[root@stripe ~]# ping 83.111.196.62 -I 85.234.115.115
PING 83.111.196.62 (83.111.196.62) from 85.234.115.115 : 56(84) bytes of data.
64 bytes from 83.111.196.62: icmp_seq=1 ttl=56 time=168 ms
64 bytes from 83.111.196.62: icmp_seq=2 ttl=56 time=178 ms
--- 83.111.196.62 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 168.441/173.542/178.644/5.118 ms

root@gizmo [~]# ping 83.111.196.59 -I 91.167.26.36
PING 83.111.196.59 (83.111.196.59) from 91.167.26.36 : 56(84) bytes of data.
--- 83.111.196.59 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1000ms

root@gizmo [~]# ping 83.111.196.60 -I 91.167.26.36
PING 83.111.196.60 (83.111.196.60) from 91.167.26.36 : 56(84) bytes of data.
64 bytes from 83.111.196.60: icmp_seq=0 ttl=56 time=159 ms
64 bytes from 83.111.196.60: icmp_seq=1 ttl=56 time=158 ms
--- 83.111.196.60 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 158.851/159.041/159.232/0.441 ms, pipe 2

root@gizmo [~]# ping 83.111.196.61 -I 91.167.26.36
PING 83.111.196.61 (83.111.196.61) from 91.167.26.36 : 56(84) bytes of data.
64 bytes from 83.111.196.61: icmp_seq=0 ttl=54 time=169 ms
64 bytes from 83.111.196.61: icmp_seq=1 ttl=54 time=178 ms
--- 83.111.196.61 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 169.092/173.624/178.156/4.532 ms, pipe 2

root@gizmo [~]# ping 83.111.196.62 -I 91.167.26.36
PING 83.111.196.62 (83.111.196.62) from 91.167.26.36 : 56(84) bytes of data.
--- 83.111.196.62 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms

root@gizmo [~]# ping 83.111.196.59 -I 91.167.26.37
PING 83.111.196.59 (83.111.196.59) from 91.167.26.37 : 56(84) bytes of data.
--- 83.111.196.59 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms

root@gizmo [~]# ping 83.111.196.60 -I 91.167.26.37
PING 83.111.196.60 (83.111.196.60) from 91.167.26.37 : 56(84) bytes of data.
64 bytes from 83.111.196.60: icmp_seq=0 ttl=57 time=149 ms
64 bytes from 83.111.196.60: icmp_seq=1 ttl=57 time=148 ms
--- 83.111.196.60 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 148.349/148.832/149.316/0.618 ms, pipe 2

root@gizmo [~]# ping 83.111.196.61 -I 91.167.26.37
PING 83.111.196.61 (83.111.196.61) from 91.167.26.37 : 56(84) bytes of data.
64 bytes from 83.111.196.61: icmp_seq=0 ttl=55 time=178 ms
64 bytes from 83.111.196.61: icmp_seq=1 ttl=55 time=178 ms

--- 83.111.196.61 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 178.416/178.474/178.532/0.058 ms, pipe 2
root@gizmo [~]# ping 83.111.196.62 -I 91.167.26.37
PING 83.111.196.62 (83.111.196.62) from 91.167.26.37 : 56(84) bytes of data.

--- 83.111.196.62 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3000ms
So sending from Stripe using interface 85.234.115.64, 83.111.196.60 and 83.111.196.61 are ok, but .59 and .62 fail. Strangely, sending from Stripe using interface 85.234.115.115 the opposite is true, .59 and .62 are ok but .60 and .61 fail! I did try both interfaces to google, and they worked ok:
Code:
[root@stripe ~]# ping www.google.com -I 85.234.115.64
PING www.l.google.com (209.85.165.147) from 85.234.115.64 : 56(84) bytes of data.
64 bytes from eo-in-f147.google.com (209.85.165.115): icmp_seq=1 ttl=240 time=112 ms
64 bytes from eo-in-f147.google.com (209.85.165.115): icmp_seq=2 ttl=240 time=114 ms

--- www.l.google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 112.559/113.535/114.512/1.032 ms
[root@stripe ~]# ping www.google.com -I 85.234.115.115
PING www.l.google.com (209.85.165.147) from 85.234.115.115 : 56(84) bytes of data.
64 bytes from eo-in-f147.google.com (209.85.165.147): icmp_seq=1 ttl=240 time=116 ms
64 bytes from eo-in-f147.google.com (209.85.165.147): icmp_seq=2 ttl=240 time=112 ms
64 bytes from eo-in-f147.google.com (209.85.165.147): icmp_seq=3 ttl=240 time=111 ms
This isnt a shorewall issue (as it does work for all ip addresses and the config is same for each alias plus Siteuptime is failing for some sites) nor is this an issue with my servers in the UK (given they work without any issue to other servers), i am sure this is a routing issue of my ISP but for the life of me i wouldnt know where to start nor how to prove this so they will fix it.

I would greatly appreciate any pointers or suggestions on the above.

Many thanks in advance,

Chris