Find the answer to your Linux question:
Results 1 to 10 of 10
Like Tree2Likes
  • 1 Post By johne
  • 1 Post By johne
HI all I've been trying to find an answer to the above and find it pretty difficult. I have one Nic on a server which is cabled directly to the ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Nov 2013
    Posts
    5

    vlans, macvlans, bridges and containers. How hard can it be?


    HI all

    I've been trying to find an answer to the above and find it pretty difficult.

    I have one Nic on a server which is cabled directly to the router, no intervening switch or hub. I have two vlans setup on the router and one open connection.

    I want to create two isolated groups of containers on the server and connect these to the router via the vlans. The host connection remaining untagged. So three subnets two of which are vlans.

    The containers also need to be isolated on the server.
    I am port forwarding to some containers so they need their own (private) IP address.

    I assume I will need two bridges (one for each group of containers)

    I assume I need to attach an outward vlan to each bridge.

    I assume I need to attach each of my container to the appropriate bridge.

    I assume I need vlan or macvlan connections between each container and its' allocated bridge.

    The problem is I don't really understand what I am doing and I am trying to hack 'vaguely related' examples on the net. Even these are hard to find. I have read lots of docs and can get part of the setup working but none of the doc's cover my complete setup and I haven't managed to fill in the gaps. I am still trying to pin down the detail on most of the basic questions:


    1) What exactly should the architecture and linkage look like?

    conainter --> macvlan --> bridge --> vlan --> host eth0 --> router
    Or

    conainter --> macvlan --> bridge(eth0) --> vlan --> router

    Or

    conainter --> vlan --> bridge(eth0) --> vlan --> router

    etc etc

    2) When setting each of these configuration up I have to answer:

    What goes in the host interface file?
    what goes in the container config file?
    what goes in the container interface file?

    3) Which components get IP addresses; bridges, vlan, both?
    Also which end of the connections? both ends? --> 4 ip's to traverse the whole path?

    4) Do I need to re-subnet as I cross the bridge ie distinct tags on the inbound and outbound links?


    I've lost count of the number of permutation I've tried and can't easily distinguess between an architectural error and procedural error or small ommission. Any help to firm up any of these issues will be greatly appreciated as it could reduce the variables and narrow my search considerably.

    I will of course post the full answer when I get there.

    Thanks in advance.

    Charlie101
    Last edited by charlie101; 11-26-2013 at 04:16 PM.

  2. #2
    Just Joined!
    Join Date
    Nov 2013
    Posts
    15
    Please see:

    wiki.debian.org/LXC/VlanNetworking


    As for the layout.

    [eth0 ] -> router (untagged)

    [ lxc-container10 -> br0.10 -> eth0.10 ] -> router (tagged vlan 10)

    [ lxc-container20 -> br0.20 -> eth0.20 ] -> router (tagged vlan 20)
    charlie101 likes this.

  3. #3
    Just Joined!
    Join Date
    Nov 2013
    Posts
    5

    Thumbs up Thanks for your help and the reference.

    Quote Originally Posted by johne View Post
    Please see:

    wiki.debian.org/LXC/VlanNetworking


    As for the layout.

    [eth0 ] -> router (untagged)

    [ lxc-container10 -> br0.10 -> eth0.10 ] -> router (tagged vlan 10)

    [ lxc-container20 -> br0.20 -> eth0.20 ] -> router (tagged vlan 20)

    Hi Johne

    Thanks for your help.
    I tried the configuration you suggested (ip on the bridge; docs ref).
    I can ping everything on the bridge both ways; subnet 1, subnet 2 and host. I can also ping the router (untagged) end point from the host. However, I can't access any router end points from the vlans (subnets) in or out.

    Conceptually I'm not even sure how to route the packets for each vlan because the router vlan end points use the high IP on each vm group's subnet:-.

    vm group A:- 10.1.1.0/28
    Router vlan A end point:- 10.1.1.14

    I haven't assigned IP's to the subnet bridge interfaces and rely on the frame forwarding at the Ethernet level. Here is my host Routing table referencing devlices rather than IP's:-

    default 10.1.0.254 0.0.0.0 UG 100 0 0 sn0
    10.1.0.0 * 255.255.255.0 U 0 0 0 sn0
    10.1.1.0 * 255.255.255.240 U 0 0 0 sn1.2
    10.1.100.0 * 255.255.255.240 U 0 0 0 sn100.1001

    10.1.0.254 is the router end point for the (untagged) host and is working for the host. Host subnet is: 10.1.0.0/24.

    But the VMs cannot ping any of the end points on the router including the host subnet router's end point 10.1.0.254. But the can ping the host interface on the bridge: 10.1.0.1 (unsurprisingly).

    If you have any ideas that would be great but you've a great help already.

    Charlie101

  4. #4
    Just Joined!
    Join Date
    Nov 2013
    Posts
    15
    I'm a bit confused by your issue. (are those snull net (sn0, sn1.2 etc) interfaces?)
    What are you trying to accomplish in the end.

    you don't add IP's to the bridge adapters, you add them to the container configuration.
    (otherwise there would be no reason to use containers).

    What type of VM's are you using?

    What kernel version are you using?

    What is your L3 configuration eg.:

    Subnet untagged: 10.1.0.0/24
    Host_IP: 10.1.0.1/24
    Router_IP: 10.1.0.254/24

    Subnet vlan A: 10.1.1.0/28
    VM_IP_A: 10.1.1.1/28
    Router__IP_VLAN_A: 10.1.1.14/28

    Subnet Vlan B: 10.1.100.0/28
    VM_IP_B: 10.1.100.1/28
    Router_IP_VLAN_B: 10.1.100.14/28


    now first add the vlan adapters to you host.

    assign the IP's to those vlans before adding any bridges and containers. check that you can ping the router vlan ip from the host.
    then unassign the IP addresses on the vlan interfaces.

    start adding the bridge and the containers.
    charlie101 likes this.

  5. #5
    Just Joined!
    Join Date
    Nov 2013
    Posts
    5
    Hi Johne

    Sorry for the delay a crisis of a different sort called me away.
    Thanks again for your help. It's really useful stepping back and testing each stage.
    I suppose I should have tried that myself but I'm too lost in the problem and too inexperienced with networking.
    So thanks again.

    Anyway; no joy I'm afraid.

    Quote Originally Posted by johne View Post
    I'm a bit confused by your issue. (are those snull net (sn0, sn1.2 etc) interfaces?)

    Answer: They are the name of my bridges containing the subnets.

    Here is the contents of the (host server) interfaces file:

    auto eth0
    iface eth0 inet manual

    # NOTE: I tried swapping the IP BETWEEN eth0 and sn0. it made not difference.

    auto sn0
    iface sn0 inet static
    bridge_ports eth0
    address 10.1.0.1
    netmask 255.255.255.0
    broadcast 10.1.0.255
    gateway 10.1.0.254
    dns_nameservers 10.0.0.254
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0


    iface eth0.2 inet manual

    auto sn1.2
    iface sn1.2 inet manual
    bridge_ports eth0.2
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0


    iface eth0.1001 inet manual

    auto sn100.1001
    iface sn100.1001 inet manual
    bridge_ports eth0.1001
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0





    What are you trying to accomplish in the end.

    Answer: I host my own web site which uses a couple of containers (LXC) to which I've been assigning individual public IPs.
    Now I want to allow a local voluntary group to use my server for their site and I want to segment the two sites as much as possible.
    I'm also moving the containers from individual public IP's to private IPs with port forwarding to free up some public IPs (/29)
    Hence the two subnets and vlans. I also believe macvlans might be useful on the host but that's a similar story.

    you don't add IP's to the bridge adapters, you add them to the container configuration.
    (otherwise there would be no reason to use containers).

    Answer: I only added the ip for the untagged device to the bridge: eth0.
    I believe I read it in one of the doc's but can't remember now.
    I also moved the IP back to eth0: no difference.

    What type of VM's are you using?

    Answer: LXC on ubuntu 12.04

    What kernel version are you using?

    Answer: 3.2.0-29-generic


    What is your L3 configuration eg.:

    Subnet untagged: 10.1.0.0/24
    Host_IP: 10.1.0.1/24
    Router_IP: 10.1.0.254/24

    Subnet vlan A: 10.1.1.0/28
    VM_IP_A: 10.1.1.1/28
    Router__IP_VLAN_A: 10.1.1.14/28

    Subnet Vlan B: 10.1.100.0/28
    VM_IP_B: 10.1.100.1/28
    Router_IP_VLAN_B: 10.1.100.14/28


    Answer: Yes, you have that correct.


    now first add the vlan adapters to you host.

    Answer: Did that.

    assign the IP's to those vlans before adding any bridges and containers. check that you can ping the router vlan ip from the host.

    Answer: Before adding the vlans I can ping the router tagged interfaces (logical vlans).
    After adding the vlans on the host server I can't ping the router vlans.


    then unassign the IP addresses on the vlan interfaces.
    start adding the bridge and the containers.

    Answer: Tried it even though previous step failed. It also failed.

    In case it's relevant my router is a pfsense 2.1
    My Server nic is
    vlan Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 09)





    These are the commands I used for creating/testing the vlans

    --------------------------------------------------->


    root@serv3:~# ifconfig
    ------------------------------------------------------------------
    eth0 Link encap:Ethernet HWaddr 50:46:5d:6b:05:31
    inet addr:10.1.0.1 Bcast:10.1.0.255 Mask:255.255.255.0
    inet6 addr: fe80::5246:5dff:fe6b:531/64 Scope:Link
    UP BROADCAST MULTICAST MTU:1500 Metric:1
    RX packets:0 errors:0 dropped:0 overruns:0 frame:0
    TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:0 (0.0 B) TX bytes:568 (568.0 B)
    Interrupt:98 Base address:0xa000

    lo Link encap:Local Loopback
    inet addr:127.0.0.1 Mask:255.0.0.0
    inet6 addr: ::1/128 Scope:Host
    UP LOOPBACK RUNNING MTU:16436 Metric:1
    RX packets:24 errors:0 dropped:0 overruns:0 frame:0
    TX packets:24 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:1980 (1.9 KB) TX bytes:1980 (1.9 KB)





    root@serv3:~# ping 10.1.0.254
    ------------------------------------------------------------
    PING 10.1.0.254 (10.1.0.254) 56(84) bytes of data.
    64 bytes from 10.1.0.254: icmp_req=1 ttl=64 time=0.460 ms
    64 bytes from 10.1.0.254: icmp_req=2 ttl=64 time=0.195 ms
    ^C
    --- 10.1.0.254 ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 999ms
    rtt min/avg/max/mdev = 0.195/0.327/0.460/0.133 ms





    root@serv3:~# ping 10.1.1.14
    ------------------------------------------------------------
    PING 10.1.1.14 (10.1.1.14) 56(84) bytes of data.
    64 bytes from 10.1.1.14: icmp_req=1 ttl=64 time=0.298 ms
    64 bytes from 10.1.1.14: icmp_req=2 ttl=64 time=0.245 ms
    ^C
    --- 10.1.1.14 ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 999ms
    rtt min/avg/max/mdev = 0.245/0.271/0.298/0.031 ms






    root@serv3:~# ping 10.1.100.14
    -----------------------------------------------------------
    PING 10.1.100.14 (10.1.100.14) 56(84) bytes of data.
    64 bytes from 10.1.100.14: icmp_req=1 ttl=64 time=0.359 ms
    64 bytes from 10.1.100.14: icmp_req=2 ttl=64 time=0.197 ms
    64 bytes from 10.1.100.14: icmp_req=3 ttl=64 time=0.220 ms
    ^C
    --- 10.1.100.14 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 1998ms
    rtt min/avg/max/mdev = 0.197/0.258/0.359/0.073 ms






    root@serv3:~# vconfig add eth0 2

    Added VLAN with VID == 2 to IF -:eth0:-


    root@serv3:~# vconfig add eth0 1001

    Added VLAN with VID == 1001 to IF -:eth0:-


    root@serv3:~# ifconfig eth0.2 up
    --------------------------------
    # Ping still works




    root@serv3:~# ip addr add 10.1.1.13/28 brd + dev eth0.2
    -------------------------------------------------------

    Ping no longer works for eth0.2:


    root@serv3:~# ping 10.1.1.14
    -----------------------------
    PING 10.1.1.14 (10.1.1.14) 56(84) bytes of data.
    ^C
    --- 10.1.1.14 ping statistics ---
    60 packets transmitted, 0 received, 100% packet loss, time 59472ms

    ----------------------------------------------------

    The only way I've successfully managed to subnet the two groups of container is by using bridges and plain veth pipes to connect them. But that doesn't give the isolation that vlans or macvlans would provide.

    I'm conscious of taking your time so if you want to bail out I quite understand. I would If I could!

    Thanks again your advice is really helpful..

    Charlie101
    Last edited by charlie101; 12-01-2013 at 03:02 AM.

  6. #6
    Just Joined!
    Join Date
    Nov 2013
    Posts
    15
    Looks like your pfsense router have untagged vlan interfaces in the other end.

    Eg. when your server tags packets leaving eth0, they will be "ignored" by the pfsense router. (eg packets are dropped) OR, there is a routing issue on your pfsense router.

    (if you are still able to ping 10.1.0.254 after adding an the vlan2 interface)


    If you are able:
    Connect a laptop directly to the server ethernet port.

    Configure the VLAN's and IP's like on the server on the laptop(linux (you can use a livedistro, like backtrack)).

    Then ping between the server and the laptop. to see if the issue is located at the pfsense router.

    eg:

    Server:

    ifconfig eth0 10.1.0.1/24 up
    vconfig add eth0 2
    ifconfig eth0.2 0.1.1.13/28 up

    Laptop:
    ifconfig eth0 10.1.0.254/24 up
    vconfig add eth0 2
    ifconfig eth0.2 10.1.1.14/28 up


    ping 10.1.0.1

    ping 10.1.1.13

  7. #7
    Just Joined!
    Join Date
    Nov 2013
    Posts
    5

    Thumbs up

    Quote Originally Posted by johne View Post
    Looks like your pfsense router have untagged vlan interfaces in the other end.

    Eg. when your server tags packets leaving eth0, they will be "ignored" by the pfsense router. (eg packets are dropped) OR, there is a routing issue on your pfsense router.

    (if you are still able to ping 10.1.0.254 after adding an the vlan2 interface)


    If you are able:
    Connect a laptop directly to the server ethernet port.

    Configure the VLAN's and IP's like on the server on the laptop(linux (you can use a livedistro, like backtrack)).

    Then ping between the server and the laptop. to see if the issue is located at the pfsense router.

    eg:

    Server:

    ifconfig eth0 10.1.0.1/24 up
    vconfig add eth0 2
    ifconfig eth0.2 0.1.1.13/28 up

    Laptop:
    ifconfig eth0 10.1.0.254/24 up
    vconfig add eth0 2
    ifconfig eth0.2 10.1.1.14/28 up


    ping 10.1.0.1

    ping 10.1.1.13
    Hi Johne

    You were right that worked!
    Something must be wrong with my pfsense box.
    As you know I'm no network expert but the GUI is pretty easy and everything I check seems OK. I will check again!

    Thanks very much for you help.

  8. #8
    Just Joined!
    Join Date
    Nov 2013
    Posts
    15
    Quote Originally Posted by charlie101 View Post
    Hi Johne

    You were right that worked!
    Something must be wrong with my pfsense box.
    As you know I'm no network expert but the GUI is pretty easy and everything I check seems OK. I will check again!

    Thanks very much for you help.
    I have never actually used pfsense.

    but i found this guide: hXXp://wwwDOTtheninjageekDOTcoDOTza/pfsense-configuring-vlans/
    (obviously replace hXXp with http and DOT with a period, i cant post links on this forum yet.)
    Please read through it and see if you might have forgotten something.

    Could possibly be that the pfsense router does not permit ping (i.e firewalled) on the VLAN interfaces.

  9. #9
    Just Joined!
    Join Date
    Nov 2013
    Posts
    5

    Thumbs up

    Quote Originally Posted by johne View Post
    I have never actually used pfsense.

    but i found this guide: hXXp://wwwDOTtheninjageekDOTcoDOTza/pfsense-configuring-vlans/
    (obviously replace hXXp with http and DOT with a period, i cant post links on this forum yet.)
    Please read through it and see if you might have forgotten something.

    Could possibly be that the pfsense router does not permit ping (i.e firewalled) on the VLAN interfaces.

    Hi Johne

    You're on the money again.
    I said the pfense gui made it easy and it does but I simply didn't look at the firewall rules for the new vlan interfaces. Which was the last point on your link above. I guess I assumed the vlans used the rules as the parent interface or I was too preoccupied with vlans and tags to think about firewall implications.

    I haven't fixed the problem yet as it's late but it is clearly an issue. I hope it's the last issue. I shall try and get on it tomorrow and let you know how it goes. Thanks again for all your help.

  10. #10
    Just Joined!
    Join Date
    Nov 2013
    Posts
    15
    Best of luck to you! let the community know if you need some help.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •