Find the answer to your Linux question:
Results 1 to 8 of 8
Hello community, I really hope anybody could give me a hand with this. It's really taking excessive time for me. Basically I am running KVM on fedora. These is my ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Aug 2012
    Posts
    6

    KVM - Really strange communication issue


    Hello community,

    I really hope anybody could give me a hand with this. It's really taking excessive time for me.

    Basically I am running KVM on fedora. These is my environment:
    qemu-kvm-1.2.2-14.fc18.x86_64 with Virtual Machine Manager 0.9.5
    Linux- 3.10.14-100.fc18.x8- x86_64 x86_64 x86_64 GNU/Linux


    I have a couple of guests running identical scientific linux:
    2.6.32-279.el6.x86_64 #1 - x86_64 x86_64 x86_64 GNU/Linux

    I am connecting both guests through a network type ' isolated' in 192.168.100.0/24

    and also to the outside world by ip forwarding and a natted connection configured in the host's iptables.

    Guest1: IP:192.168.100.2/24
    Guest2: IP: 192.168.100.7/24
    Gateway IP (to the host PC):192.168.100.1/24


    Any of the two hosts can connect perfectly to the outside world. Either telnet 4.2.2.2 53 or ping any other server outside.

    They can also see each other in their respective ARP tables and they can also ping each other!. But... They can not access to any of the mutual services.

    I think it is not an issue with the guests ip stack as they can access to their own services.
    For instance in guest1, ssh 192.168.100.2 works and in guest2, ssh 192.168.100.7 also works.

    However
    Guest 1: ssh 192.168.100.7 FAILS
    And viceversa.

    When I say ssh, I mean any other service. I try ncat in any of both servers with arbitrary ports with no luck. They are fresh installations and iptables are disabled in both guests.

    Is there any possibility of a weird bug in the KVM virtual switch or similar? My knowledge on KVM is quite limited I am afraid.

    I really need another opinion on this as I think I have spent too much time trying to troubleshoot.

    Will really really appreciate any help.

    Thanks!!

  2. #2
    Just Joined!
    Join Date
    Feb 2007
    Posts
    95
    Hello,

    I've encountered the exact same issue. I am assuming you have some type of hooks script that's forwarding IPs ssh to your guest ssh ports, correct? What's the output of "iptables -L" for your host?

    Basically, if you're using the libvirt hooks script (that's what I was using), any port that you're forwarding not be accessible for the VMs. Proof, install nmap in your virtual machines (guest) and I bet if you nmap guest1 from guest2 and vice versa on port 22 (ssh port), the connection will show as "filtered" aka firewalled. The same thing will happen if you forward any port. Let's say you forward port 80 to guest1,from that point on, your VMs will not be able to access port 80 outgoing (nc will just timeout).

    Solution: What you'll have to do, is either use the "bridge" networking instead of the default libvirt networking or actually setup a VM as router (which is what I did in my case, using vyatta) and you should be good to go (I also use bridge for some of my VMs as well). And also remove those ports from being forwarded, once you have setup bridge or a custom network for the VMs. After that, the ssh port will be accessible within the VMs. Let me know if you need more suggestions on this.

  3. #3
    Just Joined!
    Join Date
    Feb 2007
    Posts
    95
    P.S. here's the original thread I posted a few months back when I first found this problem: http://www.linuxforums.org/forum/net...port-80-a.html

    Like I said, I never really found a solution for it, other than to either use bridge network or create your own virtual router to handle your VMs network.

  4. #4
    Just Joined!
    Join Date
    Aug 2012
    Posts
    6
    Hi,

    Thanks for the immediate response. I am noy using any hook script. The connectionn to the Internet/host are dealt by the host's iptables if the destination Ip is not in the range of the isolated network (192.168.100.0/24). That's way I defined virbr1 as the default gateway in the guests.
    Anyway, the issue is with the inter-guests communication. I know I am using an isolated network with a gateway for the host but that hasn't been an issue so far. This is a new issue..
    The traffic for 192.168.100.0/24 should hit the the host's iptables. Am I right? Is like there is a firewall between my two VMs... somewhere in between but I they are the only two active VMs!


    The result of the nmap in any of the server is quite disheartening (see below) : "all ports filtered"
    Please have a look at the attched diagram and the results below and many many thanks again!!


    GUEST1
    ------
    [root@server1 ~]# netstat -nr
    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window irtt Iface
    192.168.100.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
    169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
    0.0.0.0 192.168.100.1 0.0.0.0 UG 0 0 0 eth1
    [root@server1 ~]#

    [root@server1 ~]# service iptables status
    iptables: Firewall is not running.
    [root@server1 ~]#


    [root@server1 ~]# nmap 192.168.100.7 <<<<< NMAP TO THE OTHER GUEST VM

    Starting Nmap 5.51 at 2013-10-22 00:18 BST
    Nmap scan report for 192.168.100.7
    Host is up (0.00070s latency).
    All 1000 scanned ports on 192.168.100.7 are filtered
    MAC Address: 52:54:00:4E:7B:4C (QEMU Virtual NIC)

    Nmap done: 1 IP address (1 host up) scanned in 21.21 seconds
    [root@server1 ~]#

    [root@server1 ~]# nmap 192.168.100.1 <<<<< NMAP TO THE HOST (GATEWAY IP)

    Starting Nmap 5.51 ) at 2013-10-22 00:12 BST
    Nmap scan report for 192.168.100.1
    Host is up (0.000080s latency).
    Not shown: 999 closed ports
    PORT STATE SERVICE
    22/tcp open ssh <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< My host has the port 22 opened and is accessible from the VMs
    MAC Address: 52:54:00:95:BA:26 (QEMU Virtual NIC)

    Nmap done: 1 IP address (1 host up) scanned in 0.16 seconds
    [root@server1 ~]#



    ------------------------------------------------

    GUEST2
    ------
    [root@tester1 ~]# service iptables status
    iptables: Firewall is not running.
    [root@tester1 ~]#

    [root@tester1 ~]# netstat -nr
    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window irtt Iface
    192.168.100.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
    169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth1
    0.0.0.0 192.168.100.1 0.0.0.0 UG 0 0 0 eth1
    [root@tester1 ~]#

    [root@tester1 ~]# nmap 192.168.100.2

    Starting Nmap 5.51 at 2013-10-22 00:22 BST
    Nmap scan report for 192.168.100.2
    Host is up (0.00073s latency).
    All 1000 scanned ports on 192.168.100.2 are filtered <<<<< NMAP TO THE OTHER GUEST VM
    MAC Address: 52:54:00:25:06:7A (QEMU Virtual NIC)

    Nmap done: 1 IP address (1 host up) scanned in 27.74 seconds
    [root@tester1 ~]#

    [root@tester1 ~]# nmap 192.168.100.1

    Starting Nmap 5.51 at 2013-10-22 00:23 BST
    Nmap scan report for 192.168.100.1 <<<<< NMAP TO THE HOST (GATEWAY IP)
    Host is up (0.000070s latency).
    Not shown: 999 closed ports
    PORT STATE SERVICE
    22/tcp open ssh
    MAC Address: 52:54:00:95:BA:26 (QEMU Virtual NIC)

    Nmap done: 1 IP address (1 host up) scanned in 6.74 seconds
    [root@tester1 ~]#




    ---------------------------------------------------------
    HOST:

    [root@jotasanfed ~]# netstat -nr
    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window irtt Iface
    0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 eth1
    172.16.100.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr3
    192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
    192.168.100.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr1
    192.168.200.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
    192.168.220.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr2
    [root@jotasanfed ~]#

    [root@jotasanfed ~]# iptables
    iptables v1.4.16.2: no command specified
    Try `iptables -h' or 'iptables --help' for more information.

    [root@jotasanfed ~]# cat /etc/sysconfig/iptables
    # Generated by iptables-save v1.4.14 on Thu Mar 28 23:17:30 2013
    *nat
    :PREROUTING ACCEPT [586:56928]
    :INPUT ACCEPT [4:336]
    :OUTPUT ACCEPT [1113:69238]
    :POSTROUTING ACCEPT [10:950]
    -A POSTROUTING -o eth1 -j MASQUERADE
    COMMIT
    # Completed on Thu Mar 28 23:17:30 2013
    # Generated by iptables-save v1.4.14 on Thu Mar 28 23:17:30 2013
    *filter
    :INPUT ACCEPT [0:0]
    :FORWARD ACCEPT [8:2196]
    :OUTPUT ACCEPT [1321:341262]


    -A FORWARD -i eth1 -o virbr1 -m state --state RELATED,ESTABLISHED -j ACCEPT
    -A FORWARD -i eth1 -o virbr1 -j ACCEPT
    -A FORWARD -i virbr1 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
    -A FORWARD -i virbr1 -o eth1 -j ACCEPT
    #-A FORWARD -i em1 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
    #-A FORWARD -i em1 -o eth1 -j ACCEPT
    -A FORWARD -p icmp -m icmp --icmp-type 8 -j ACCEPT
    #-A FORWARD -o virbr1 -i eth1 -j ACCEPT
    -A FORWARD -p icmp -m icmp --icmp-type 0 -j ACCEPT


    -A FORWARD -o virbr0 -i eth1 -j ACCEPT
    -A FORWARD -i eth1 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT

    -A FORWARD -i eth1 -o em1 -j ACCEPT
    -A FORWARD -i eth1 -o em1 -m state --state RELATED,ESTABLISHED -j ACCEPT


    -A FORWARD -i virbr0 -o eth1 -j ACCEPT
    -A FORWARD -j DROP

    -A INPUT -p udp --sport 123 -j ACCEPT
    -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
    -A INPUT -p icmp -j ACCEPT
    -A INPUT -i lo -j ACCEPT
    -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
    -A INPUT -m limit --limit 15/minute -j LOG --log-level 7 --log-prefix "Dropped by firewall: "
    -A INPUT -p icmp -m icmp --icmp-type 3 -j REJECT --reject-with icmp-host-prohibited
    -A INPUT -p icmp -m icmp --icmp-type 10 -j REJECT --reject-with icmp-host-prohibited
    -A INPUT -p icmp -m icmp --icmp-type 4 -j REJECT --reject-with icmp-host-prohibited
    -A INPUT -p icmp -m icmp --icmp-type 11 -j REJECT --reject-with icmp-host-prohibited
    -A INPUT -p tcp -j REJECT --reject-with tcp-reset
    -A INPUT -p udp -j REJECT --reject-with icmp-port-unreachable

    COMMIT
    # Completed on Thu Mar 28 23:17:30 2013

    [root@jotasanfed ~]#

    --------------------------
    And this is what I see by tcpdump in the virbr1 from the host while trying to ssh from guest1 to guest2, only packets in one direction, no responses:

    root@jotasanfed ~]# tcpdump -nni virbr1 host 192.168.100.7 or host 192.168.100.2
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on virbr1, link-type EN10MB (Ethernet), capture size 65535 bytes
    10:03:39.507528 IP 192.168.100.7.34776 > 192.168.100.2.22: Flags [S], seq 3219089940, win 14600, options [mss 1460,sackOK,TS val 6350812 ecr 0,nop,wscale 6], length 0
    10:03:41.507525 ARP, Request who-has 192.168.100.2 tell 192.168.100.7, length 28
    10:03:41.508093 ARP, Reply 192.168.100.2 is-at 52:54:00:25:06:7a, length 28
    10:03:43.507586 IP 192.168.100.7.34776 > 192.168.100.2.22: Flags [S], seq 3219089940, win 14600, options [mss 1460,sackOK,TS val 6354812 ecr 0,nop,wscale 6], length 0
    10:03:51.507544 IP 192.168.100.7.34776 > 192.168.100.2.22: Flags [S], seq 3219089940, win 14600, options [mss 1460,sackOK,TS val 6362812 ecr 0,nop,wscale 6], length 0
    10:04:07.507606 IP 192.168.100.7.34776 > 192.168.100.2.22: Flags [S], seq 3219089940, win 14600, options [mss 1460,sackOK,TS val 6378812 ecr 0,nop,wscale 6], length 0
    Attached Images Attached Images

  5. #5
    Just Joined!
    Join Date
    Feb 2007
    Posts
    95
    Like you have mentioned, your nmap from vm to another definitively reveals something weird going on. In my case, it was due to the hooks script, but from the looks of it, you're only using NAT to give your VMs access to the internet, which you said you were able to ping 4.2.2.1, so at least that part is working.

    Although, one thing that I'll let you know (at least with the default libvirt setup), is that libvirt adds it's own set of rules there in iptables (for the host). Any changes that you make to the host /etc/sysconfig/iptables is usually overwritten by libvirt (and thus the need for a hooks script for destination port forwarding).

    My recommendation is that run the following commands on the host:

    -----

    # iptables -F (this is gonna flush all the rules)
    # /etc/init.d/iptables stop (stop iptables)
    # /etc/init.d/libvirtd stop (stop libvirt)
    # /etc/init.d/iptables start
    # /etc/init.d/libvirtd start
    -----

    After you run all of that, run "iptables -L" to list all of the rules and see if they have changed by any chance. And also check ssh between the VMs to see if anything changed.

  6. #6
    Linux Engineer
    Join Date
    Apr 2012
    Location
    Virginia, USA
    Posts
    795
    Most likely IP tables on your host machine. If you set up the NATing by hand, it's possible you made an error or omission. Further more, both NICs have to be part of the same virtual network. It's not enough to just give them both IPs in the same subnet if they're not on the same virtual network. What we need to see if your host's virtual network settings, not your guest's settings.

  7. #7
    Just Joined!
    Join Date
    Aug 2012
    Posts
    6
    Hi,
    I have followed bigb89 commands and not it works!.

    mizzle, I am using an network of the type 'isolated' (defined through the virtual manager). This is its xml
    [root@jotasanfed networks]# cat interfaz.xml
    <!--
    WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
    OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
    virsh net-edit interfaz
    or other application using the libvirt API.
    -->

    <network>
    <name>interfaz</name>
    <uuid>693db6fd-103f-e213-e33d-0542d7980b67</uuid>
    <bridge name='virbr1' stp='on' delay='0' />
    <mac address='52:54:00:95:BA:26'/>
    <ip address='192.168.100.1' netmask='255.255.255.0'>
    </ip>

    Anyway, now irt works (thanks!) but I will keep a close eye to see if it reoccurs.

    Cheers!
    Last edited by jotasan; 10-23-2013 at 09:44 PM.

  8. #8
    Just Joined!
    Join Date
    Feb 2007
    Posts
    95
    Hello jotasan, great to hear everything is working after the commands.

    It's very rare, but with iptables and libvirt, sometimes they go out of sync (somewhat) and you'll need to run those commands in order to fix connectivity.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •