NAT rule not working again after server reboot

[juchu]

Hi guys,

I'm newbie of linux networking and have been try and error for few days, without any progress.....any help would be much appreciated!!! Thanks guys!!!

The background:

• A physical server running CentOS 6 with KVM installed, 2 vm is defined now, named as VM 2 and VM 3 and ip is 192.168.122.2 and 192.168.122.3 accordingly
• Iptables running at host OS but NOT at guest OSs.
• Host OS iptables has configure NAT as follow:
  
• 2 DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:30022 to:192.168.122.3:22
• 3 DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:20022 to:192.168.122.2:22
• Thus this is expected to have host OS high port to be forward to VM's ssh port

The situation:

• Everything work fine after installation, and we have run for a month, however after a RAM upgrade at physical server - that means the server is shutdown and reboot, the NAT/port forward is not function anymore, given that:
• Host OS iptables is started, while guest OS is keep as stop
• Host OS can ssh/telnet VM's ssh port (22)
• Result of command iptables -t nat -L -v -n:
  
• pkts bytes target prot opt in out source destination
• 6 304 LOG tcp -- br0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:30022 LOG flags 0 level 4 prefix `ROUTE APP3'
• 6 304 DNAT tcp -- br0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:30022 to:192.168.122.3:22
• 3 152 DNAT tcp -- br0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:20022 to:192.168.122.2:22
• Package is handled by the NAT rule while I try to telnet 30022 from my local PC (separate network)

Any ideas..??

[atreyu]

Do you have to enable IP forwarding in the kernel? e.g.:

echo 1 > /proc/sys/net/ipv4/ip_forward

If that is it, you can put it in /etc/sysctl.conf:

net.ipv4.ip_forward = 1

More here.

[juchu]

Thanks for your reply! But I'm afraid I have already enable the ip forward:

# cat /proc/sys/net/ipv4/ip_forward
# 1


If there is something do with the reboot, I suspect there are some services unexpectedly start/stop...but then other then iptables/ip forward...I didn't think of any other possibility.

[atreyu]

buggah...i thought that was it for sure. Is the SSH daemon running on the two VM machines I assume?

Have you tried using SSH to connect, using the -'v' flag?

[juchu]

Yes, both VM SSH daemon is running, this is proved as i can connect them from HOST OS by ssh command.

I didn't try before, this is the return when i execute command at another linux machine:

[root@ad-a19-29 ~]# ssh -v -p 30022 <public ip>
OpenSSH_4.3p2, OpenSSL 0.9.8b 04 May 2006
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to <public IP> [<public IP>] port 30022.
debug1: connect to address <public IP> port 30022: Connection refused
ssh: connect to host <public IP> port 30022: Connection refused

p.s. sorry for masking those sensitive data, the <public IP> is 202.xxx.xxx.xxx, its my real ip :P

[atreyu]

So back to iptables...can you post your iptables config file? I know you said all was fine before, and the problem is due to a system reboot, but who knows, maybe a bit slipped in there somewhere when you weren't looking.

Also, SELinux is not running on the main server, is it?

[juchu]

Yes you are right, everything is possible Here is the iptables config:

[root@mhost network-scripts]# cat /etc/sysconfig/iptables
# Generated by iptables-save v1.4.7 on Tue Nov 22 13:22:33 2011
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [172:28850]
-A INPUT -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5907 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 30022 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -s 192.168.122.1/32 -p icmp -j LOG --log-prefix "FORWARD virbr0"
-A FORWARD -d 192.168.122.1/32 -p icmp -j LOG --log-prefix "FORWARD virbr0"
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -m physdev --physdev-is-bridged -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i br0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j ACCEPT
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Tue Nov 22 13:22:33 2011
# Generated by iptables-save v1.4.7 on Tue Nov 22 13:22:33 2011
*mangle
:PREROUTING ACCEPT [401:29557]
:INPUT ACCEPT [208:20281]
:FORWARD ACCEPT [7:384]
:OUTPUT ACCEPT [172:28850]
:POSTROUTING ACCEPT [172:28850]
-A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Tue Nov 22 13:22:33 2011
# Generated by iptables-save v1.4.7 on Tue Nov 22 13:22:33 2011
*nat
:PREROUTING ACCEPT [7:332]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A PREROUTING -i br0 -p tcp -m tcp --dport 30022 -j LOG --log-prefix "ROUTE APP3"
-A PREROUTING -i br0 -p tcp -m tcp --dport 30022 -j DNAT --to-destination 192.168.122.3:22
-A PREROUTING -i br0 -p tcp -m tcp --dport 20022 -j DNAT --to-destination 192.168.122.2:22
-A PREROUTING -i br0 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.122.3:80
-A PREROUTING -i br0 -p tcp -m tcp --dport 34434 -j DNAT --to-destination 192.168.122.3:4434
-A PREROUTING -i br0 -p tcp -m tcp --dport 34435 -j DNAT --to-destination 192.168.122.3:18080
-A PREROUTING -i br0 -p tcp -m tcp --dport 23306 -j DNAT --to-destination 192.168.122.2:3306
-A PREROUTING -i br0 -p tcp -m tcp --dport 37999 -j DNAT --to-destination 192.168.122.3:7999
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p icmp -j MASQUERADE
COMMIT
# Completed on Tue Nov 22 13:22:33 2011

where the ifconfig is:

[root@mhost network-scripts]# ifconfig
br0       Link encap:Ethernet  HWaddr D0:27:88:6D:70:E9
          inet addr:<public IP>  Bcast:<public IP:255>  Mask:255.255.255.0
          .........

eth0      Link encap:Ethernet  HWaddr D0:27:88:6D:70:E9
          inet6 addr: fe80::d227:88ff:fe6d:70e9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:150606 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18233 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:11176760 (10.6 MiB)  TX bytes:4131431 (3.9 MiB)
          Interrupt:27 Base address:0x6000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:52 errors:0 dropped:0 overruns:0 frame:0
          TX packets:52 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4898 (4.7 KiB)  TX bytes:4898 (4.7 KiB)

virbr0    Link encap:Ethernet  HWaddr FE:54:00:46:6F:5F
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1097 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1574 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:279054 (272.5 KiB)  TX bytes:153651 (150.0 KiB)

vnet0     Link encap:Ethernet  HWaddr FE:54:00:75:11:C0
          inet6 addr: fe80::fc54:ff:fe75:11c0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:19 errors:0 dropped:0 overruns:0 frame:0
          TX packets:26009 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:1227 (1.1 KiB)  TX bytes:1350380 (1.2 MiB)

vnet1     Link encap:Ethernet  HWaddr FE:54:00:46:6F:5F
          inet6 addr: fe80::fc54:ff:fe46:6f5f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:825 errors:0 dropped:0 overruns:0 frame:0
          TX packets:27209 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:257644 (251.6 KiB)  TX bytes:1460966 (1.3 MiB)

And finally.........................I just realize this! I guess you have already point me to the right direction! the SELinux is on now! I suppose I have set it as disable before reboot:

SELINUX=enforcing
SELINUXTYPE=targeted

So suppose it should work when i disable and reboot...? Going to try now.

[atreyu]

you get do quick test:

setenforce 0

[juchu]

Um...bad news, either use setenforce 0, or edit /etc/sysconfig/selinux to update SELINUX=disabled, after reboot (again), same situation: Host OS can ssh to guest OS, but outside the network it return Connection refused.

[atreyu]

Okay, is it possible that there were iptables rules set up dynamically at some point while the main server was up, and these rules were never written to the system configuration file? That's the only other thing I can think of.

I did not attempt to parse your iptables config file, btw, that is a biggie. Any iptables gurus out there?