Using multiple network cards in XEN 3.0
Posted by itsec on Tue 5 Dec 2006 at 11:04
Xen is great. But installing more than one network card became a pain when I tried it the first time. There are some documents describing the principle but I was unable to find a real life example somewhere else. So this is a summary about how it works here now.
Using a bridge for a Dom is generally a good idea but then all packets traversing the bridge can be intercepted by any Dom that is using the same bridge. Having a single network card in a Xen landscape also means that theoretically each Dom would be able to sniff all packets traversing this single network card including packets to and from other Doms. A solution is to have more than one network card attached to Xen using a single network card for a single dom.
The scenario described here has a server with 3 network cards installed. The first card should be used to access Dom0 and some other DomNs while the second and third network card should be used to purely access Dom1 rsp. Dom2. The Dom configuration file just needs to select the appropriate bridge for each dom.
Topology:eth0 - xenbr0 - Dom0, DomN eth1 - xenbr1 - Dom1 (cannot be sniffed by Dom0, DomN or Dom2) eth2 - xenbr2 - Dom2 (cannot be sniffed by Dom0, Dom1 or Domn)
The benefit of using bridging is that no manual routing configuration is required as all routes are dealt with by Xen itself.
/etc/xen/xend-config.sxp:... #(network-script network-bridge) (network-script my-network-script) ...
Change the networking to have more than a single bridge. Here we set up a new script that will start a bridge for each NIC installed:
/etc/xen/scripts/my-network-script:#!/bin/sh dir=$(dirname "$0") "$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=xenbr0 "$dir/network-bridge" "$@" vifnum=1 netdev=eth1 bridge=xenbr1 "$dir/network-bridge" "$@" vifnum=2 netdev=eth2 bridge=xenbr2Do not forget to chmod u+x this script!
And finally this is how each DomU can be configured:
/etc/xen/anyXmDomain.cfg:Change IP and MAC as YOU need it!... # use eth0 for this DomU vif = ['ip=10.XX.XX.230,mac=00:17:de:ad:fa:ce,bridge=xenbr0'] ...or... # use eth1 for Dom1 vif = ['ip=10.XX.XX.234,mac=00:16:de:ad:be:ef,bridge=xenbr1'] ...or... # use eth2 for Dom2 vif = ['ip=10.XX.XX.238,mac=00:18:be:ef:fa:cd,bridge=xenbr2'] ...
As said, there is no additional routing required in Dom0 or in DomU besides just normal routing as you would do with a single network card attached to Xen.
From DomU perspective nothing changes. Each DomU will automatically use the bridge defined in the configuration file. The only change in behavior you will notice is that the LEDs of the second and third NIC start blinking as soon as Dom1 rsp. Dom2 send or receive packets. You can even pull out the cable from the first NIC (eth0) while Dom1 (eth1) and Dom2 (eth2) continue working normally.
Dom0 routing:
# netstat -arn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 10.XX.XX.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 0.0.0.0 10.XX.XX.254 0.0.0.0 UG 0 0 0 eth0
The script above will create these bridges automatically for you so there is no need to manually change anything in the bridging settings.
Dom0 bridging:
# brctl show
bridge name bridge id STP enabled interfaces
xenbr0 8000.feffffffffff no peth0
vif0.0
vif1.0
vif3.0
vif4.0
xenbr1 8000.feffffffffff no peth1
vif0.1
vif6.0
xenbr2 8000.feffffffffff no peth2
vif0.2
vif7.0
Each DomU can be used as usual. The DomU itself is not even aware that it is using another Xen bridge. From DomUs point of view there is a (virtual) NIC that will be used as eth0.
Dom1/Dom2 eth0 configuration: (HWaddr is Dom1)
# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:16:DE:ED:BE:EF
inet addr:10.XX.XX.234 Bcast:10.XX.XX.255 Mask:255.255.255.0
inet6 addr: fe80::216:daff:feda:ba5e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:257357 errors:0 dropped:0 overruns:0 frame:0
TX packets:238053 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:32954128 (31.4 MiB) TX bytes:51239288 (48.8 MiB)
There is nothing special about DomU routing. As it does not know about the Xen bridge it routes normally to the gateway which is 10.XX.XX.254 in this example.
Dom1/Dom2 routing:
Well, that is mainly IT. Easy! Starting any DomU now will use the appropriate interface. So each interface takes the full benefit from individual bridging. From my point of view this is a much better approach than to control the pci interface directly from DomX which would also be possible.# netstat -arn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 10.XX.XX.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 0.0.0.0 10.XX.XX.254 0.0.0.0 UG 0 0 0 eth0
If I forgot to mention something or you have corrections please give me a friendly hint.
Cheers and have fun,
Torsten
[ Parent | Reply to this comment ]
A very good question. I logged into my Dom1 and tried:
ifconfig -a | grep -i PROM
and got an empty result.
Did the same in DOM0 and also got an empty result. Seems my system is not using promiscious mode at all... ??
[ Parent | Reply to this comment ]
;-))
/Daniel
--
Ever noticed something? Unix comes with compilers. Windows comes with
Solitaire.
[ Parent | Reply to this comment ]
--
Roberto C. Sanchez
http://people.connexer.com/~roberto
[ Parent | Reply to this comment ]
As I did not try it I cannot be sure but specifying multiple entries in the /etc/xen/anyXmDomain.cfg:entry should do that:
vip=['ip=10.XX.XX.230,mac=00:16:de:ad:fa:ce,bridge=xenbr0', 'ip=10.XX.XX.234,mac=00:16:de:ad:be:ef,bridge=xenbr1']
As said, I did not try it and wrote the syntax from memory...
[ Parent | Reply to this comment ]
This was exactly what I've been thinking about for a couple of days.
I guess that this will also work with VLAN tagged interfaces?
Do you know if there's a limit on how many bridged interfaces Xen can support? I want to do a crazy setup with about 48 virtual machines each bridged to a VLAN tagged interface.
/Daniel
--
Ever noticed something? Unix comes with compilers. Windows comes with
Solitaire.
[ Parent | Reply to this comment ]
as I never used vlan tagged interfaces (and never would) I cannot answer your question. But I would ask you to write an article or a comment as soon as you figured it out. Thank you.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
that was exactly what I was looking for. This article solves me hours of trial and error.
[ Parent | Reply to this comment ]
I have a doubt about it. In the Dom0 you need to have the interfaces in the way to connect them to the bridges. In may case I have eth0 with an its network configuration and I don't want to touch it or make a bridge. Howeber, I have an eth1 that I would like to use it in the DomX. My problem is that in Dom0 I need to configure it, because not, I cannot make the bridge. So my question is how I "configure" without an IP a eth in Dom0 to use it in DomX?
Regards,
Leo
[ Parent | Reply to this comment ]
in my configuration I created bridges for all ethX cards so your configuration differs from what we have here.
What happens when you change the script /etc/xen/scripts/my-network-script:
and use only
it should create
xenbr1 should be usable as desribed in the article. What do you do with eth0 and why don't you want to also bridge it?
regards,
Torsten
[ Parent | Reply to this comment ]
>in my configuration I created bridges for all ethX cards so your configuration
>differs from what we have here.
ok, no problem, but I don't want to assign an IP to the eth1 because I shouldn't want that Dom0 have any relation in a network level with eth1.
> What happens when you change the script /etc/xen/scripts/my-network-script:
> |#!/bin/sh
> |dir=$(dirname "$0") "
> |$dir/network-bridge" "
> |$@" vifnum=1 netdev=eth1 bridge=xenbr1|
is created, but when I try to use it I have:
bridge name bridge id STP enabled interfaces
xenbr0 8000.feffffffffff no vif0.0 peth0
xenbr1 8000.000000000000 no
>and use only vif = ['ip=10.XX.XX.234,mac=00:16:de:ad:be:ef,bridge=xenbr1']
>it should create xenbr1 8000.feffffffffff no peth1 vif0.1 vif6.0
>xenbr1 should be usable as desribed in the article.
>What do you do with eth0 and why don't you want to also bridge it?
Ok, I have to explain better. I have a server that is working and running on Dom0, It has its own network interface eth0 and is working. I would like to have a DomU box, running inside for experimental proposes with its own eth interface connecting to another network. I don't want any possible interference betwend Dom0 and DomU, so I prefer to separate the network transit. That's my idea.
Howeber I'm not be able to do that if I don't configure eth1 in Dom0, and this is what I don't want to do.
Regards,
Leo
[ Parent | Reply to this comment ]
sorry but I never tried that. Needless to say that runing any server in Dom0 is an improper setup for a XEN system.
> I don't want any possible interference betwend Dom0 and DomU,
As Dom0 is always the parent of DomU that is just not possible! If your server were in a DomU it would be no problem. But running services in Dom0 gives them all privileges.
xenbr1 does not have an IP address in Dom0. Was that what you really wanted?
regards,
Torsten
[ Parent | Reply to this comment ]
>sorry but I never tried that. Needless to say that runing any server in Dom0 >is an improper setup for a XEN system.
Yes, you are right, but this is what I have by now.
>> I don't want any possible interference betwend Dom0 and DomU,
>As Dom0 is always the parent of DomU that is just not possible! If your
>server were in a DomU it would be no problem. But running services in Dom0
>gives them all privileges.
Yes, ok by now in my case, it's just an experiment. Dom0 is ok for my. I don't want that DomU and Dom0 have interferences.
> xenbr1 does not have an IP address in Dom0. Was that what you really wanted?
I want that, as you when I run brctl to have:
$brctl show
bridge name bridge id STP enabled interfaces
xenbr0 8000.feffffffffff no vif0.0
peth0
xenbr1 8000.000000000000 no
because in xenbr1 nothing is attached to eth1 because eth1 is not configured. The question to me is what I have to put in /etc/network/interfaces to configure eth1 in Dom0 without an IP or any network option to use it in DomU?
I want to use eth1 _only_ in DomU, not in Dom0.
Regards,
Leo
[ Parent | Reply to this comment ]
I am sure you are aware of this, but the xen network script(xen-bridge) creates two bridges with the same Hardware address, and messages like the followig are showing my in the logs:
Jul 9 17:15:25 salerno kernel: peth0: received packet with own address as source address
Jul 9 17:19:38 salerno kernel: peth0: received packet with own address as source address
Jul 9 17:19:38 salerno kernel: peth0: received packet with own address as source address
Jul 9 17:19:42 salerno kernel: peth0: received packet with own address as source address
Jul 9 17:30:27 salerno kernel: peth0: received packet with own address as source address
Jul 9 17:30:27 salerno kernel: peth0: received packet with own address as source address
Jul 10 09:12:19 salerno kernel: peth0: received packet with own address as source address
Jul 10 10:06:04 salerno kernel: peth0: received packet with own address as source address
Jul 10 10:06:04 salerno kernel: peth0: received packet with own address as source address
Jul 10 10:06:05 salerno kernel: peth0: received packet with own address as source address
Jul 10 10:50:33 salerno kernel: peth0 adding interface with same address as a received packet
While eth0 and eth1 have different hardware addresses(therefore all packets will endup in the host vm), I wonder how the hypvisor will handle the packets between the multiple guest oses. I am already seeing degradation in performance. Let me know what you think.
--
uk
ukhanlists AT hotmail.com
[ Parent | Reply to this comment ]
As per the article, I created my-network-script in /etc/xen/scripts with the same content and modified the existing /etc/xe/xend-config.sxp (to use my script) as below
(network-script my-network-script)
I also did "chmod +x my-network-script"
Now, if I issue "xend restart" , my Dom0 just hangs. Is there some problem with my-network-script ?? Because with /etc/xen/scripts/network-bridge, I can successfully create a DomU.
Please reply fast.
-Kaushik
[ Parent | Reply to this comment ]
"$dir/network-bridge" "$@" vifnum=0 bridge=xenbr0
instead of
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=xenbr0
in /etc/xen/scripts/my-network-script and "xend restart" worked (I can see xen-br0 apart from xenbr0, xenbr1, xenbr2 created by my-network-script) but why did it use to hang with "netdev=eth0" ?? I havent yet tried creating a domU as listed in this article...
[ Parent | Reply to this comment ]
Fernando
[ Parent | Reply to this comment ]
if i have 2 NIC eth0 and eth1 configured to 2 networks ip, i want all domU have 2 interface (2 ip address or network), how to configure the system? Any help will be appreciated!!
Thanks,
Lee
[ Parent | Reply to this comment ]
regards,
Torsten
[ Parent | Reply to this comment ]
I created 3VMs, each one using a distinct NIC.
Everything has been working fine, the traffic of each VM using its own NIC, UNTIL I REBOOTED!
One of the VM was then unable to launch (xm create...).
I had to rebuild that VM (xm-create-image...)
I could then load that VM and the 2 others, UNTIL I REBOOTED (goto to line 4 above)... So I can't rely on that setup.
Apparently bridges created that way are not stable and they don't survive a reboot!
Read somewhere that netloop device had to be at least 32 for as many as 4 NICs, raised it to 32 (in /etc/modprobe.d/netloop) but didn't change anything.
All 4 NIC work fine in the dom0! can ping in and out...
[ Parent | Reply to this comment ]
check /var/log/xen/xend.log (and the other logfiles) to see if you can find any trail of an error when booting up. As I never had 4 NICs I don't know about that.
I know that the configuration works fine using 3 NICs on at least 3 machines in 3 countries. On these machines the bridges work fine for more than a year now.
Please recheck your configuration and logs.
regards,
Torsten
[ Parent | Reply to this comment ]
Finding an explanation (not exhaustive) and even better, a solution!
From various sources in forums I could see bits of explanation but none could solve the problem.
With my forth NIC I apparently had reached the number of netloop.
There is still a confusion in my mind between loop, netloop and loopbacks (if someone could give a clear explanation of similarities, differences between these notions, I'd appreciate) but to solve the problem I had to create a file /etc/modprobe.d/netloop and put that line in it:
options netloop nloopbacks=32
(put a line feed after the line, as we have to do in /etc/fstab)
I then tried to put 6 NICS in the machine created 5 VMs. I could load 4 of them, because I had reached a new limit.
I then added the line:
loop max_loop=64
in the /etc/modules file
After a reboot, I could then load all the 5 VM, each using its NIC (eth1 to 5), dom0 still capable of using eth0...
That's still empirical! If someone could please give us the mathematic formula to set loop, netloop, loopbacks according to the number of real NICs we want to bridge...
The recipe above works fine for my older computers (read «those whose hardware is supported by Debian Etch»).
- - - - - -
But I have a brand new to setup with the same pattern (1 NIC/VM) and I had to use Ubuntu 8.0.4 (hardy) to get appropriate support for chipset (Intel G33) and Marvell Yukon II gigabit Ethernet. Xen bridges have apparently been renamed in their xen package (they now bear the physical interface name eth0 is eth0's bridge!). That new naming will probably show up in Lenny as soon as Xen kernels will be released for it! (don't know how to check xen and xen-tools version number used)
With the «my-network-script ;» proposed by Itsec, eth devices didn't even show up at boot!
I tried to adapt it, as follow:
#!/bin/sh
dir=$(dirname "$0")
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=eth0
"$dir/network-bridge" "$@" vifnum=1 netdev=eth1 bridge=eth1
"$dir/network-bridge" "$@" vifnum=2 netdev=eth2 bridge=eth2
At least, the eth0, eth1, eth3 showed up and I can ping other machines, but I still can't access internet and did'nt even try to create VMs.
Can somebody adapt the script
[ Parent | Reply to this comment ]
My DNS server is on the 192.168.100.0 network. In the first DomU, I could ping local machines and the Internet. On the second DomU, I could ping local machines, but could not get to the Internet. I had to setup, on the second DomU, to use the DNS server on the other subnet. Once I did that, both machines could access the Internet.
[ Parent | Reply to this comment ]
If you want to use this in Debian Lenny, just do everything the same except that you only supply the netdev-Variable to the network-bridge script, the other two(vifnum, bridge) are no longer needed.
And you have to specify bridge=ethX in your Domains vif-config-line, not bridge=xenbrX as in etch.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
http://debiangeeks.blogspot.com/2008/03/howto-debian-xen-e-vlan.h tml the author explain how to manage vlans in xen. I think the way they used was the simplest one. In fact, using a customized script as you have shown, if you need to add a new vlan bridge you should edit both /etc/network/interfaces and the script itself while in that post they simply edit the "interfaces" file.
I have tried it and it works!
[ Parent | Reply to this comment ]
http://www.spirit.com/Network/net0103.html
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
thanks for debian-administration very nice resource!
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
Finally I use this article myself to remember the configuration ;-)
regards,
Torsten
[ Parent | Reply to this comment ]
#!/bin/sh
dir=$(dirname "$0")
"$dir/network-bridge" "$@" netdev=eth0
"$dir/network-bridge" "$@" netdev=eth1
"$dir/network-bridge" "$@" netdev=eth2
"$dir/network-bridge" "$@" netdev=eth3
[ Parent | Reply to this comment ]