ACC SHELL
XEN/KVM networking with sysconfig (SLE-11 SP1)
==============================================
(by Marius Tomaschewski <mt@suse.de>)
This document attempts to describe basic scenarios and configuration steps
required to use sysconfig alias ifup mechanisms to configure networking on
a XEN/KVM host system. The sysconfig side of the KVM setup we describe here
is basically the same as for XEN.
1) Bridged
1.1) Bridged to a physical network
The most common scenario is to configure a bridge and connect the virtual
machines directly to the network behind the physical interface adding the
physical interface as well as the interfaces of the virtual machines to
the bridge.
This configuration can be illustrated as two switches connected together.
The physical ethernet network interface (eth0) can be considered as the
"up-link" port connecting the bridge switch and the physical network
switch, the virtual bridge interface (br0) as the network interface of
the host system plugged into a port on the bridge switch and allowing
to set up IP addresses for the host machine and also to manage the
bridge switch.
+---------------------+
| HOST / dom0 | {other hosts}
| | . .
| <br0> <eth0>[X]-------+ . .
| | | | | | |
| +-[X]---...---[X]-+ | +-[X]-[X]-...-[X]-+
| | bridge switch | | | physical |
| +-[X]---...---[X]-+ | | network switch |
| | | | +-[ ]-[ ]-...-[ ]-+
| <vif1.0> <vifN.0> |
| | | |
| +-[X]--+ +--[X]-+ |
| | dom1 | | domN | |
| +------+ +------+ |
+---------------------+
Bridged networking is supported by yast2 network on SLE-11 and openSUSE
11.1 and is proposed by default to the user at installation time, as soon
as virtualization software like XEN, KVM or Virtual Box is selected for
the installation.
The steps done by the yast2 network proposal are:
- create a bridge for each physical ethernet card
- add each physical ethernet card to the bridge
- use the bridge interface instead of the ethernet card
interface for IP configuration of the host machine
To use the configuration for a virtual machine e.g. in virt-manager:
- select a bridge interface for each virtual machine interface
Bridges are auto-detected by the SuSEfirewall2 and bridged traffic is
allowed by default -- see also the SuSEfirewall2 variable
FW_FORWARD_ALLOW_BRIDGING
in /etc/sysconfig/SuSEfirewall2. The SuSEfirewall2 script implements it
by adding the following rule for IPv4 and IPv6:
$iptables -A FORWARD -m physdev --physdev-is-bridged -j ACCEPT
The network configuration inside of the virtual machine is equivalent
to configuration of a normal ethernet interface of a host, because it
is bridged directly to the physical network and can use same services
as the host machine.
Example:
- Network configuration in the host system:
/etc/sysconfig/network/ifcfg-eth0:
STARTMODE='auto'
BOOTPROTO='none'
/etc/sysconfig/network/ifcfg-br0:
STARTMODE='auto'
BOOTPROTO='static'
# The boot protocol can be also 'dhcp'.
# In case of 'static' the IP static
# configuration:
IPADDR='192.168.0.1/24'
IPADDR_1='2001:db8:abba::1/64'
# bridge settings
BRIDGE='yes'
BRIDGE_PORTS='eth0'
BRIDGE_STP='off'
BRIDGE_FORWARDDELAY='0'
/etc/sysconfig/network/routes (or ifroute-br0)
# IPv4
default 192.168.0.254 - br0
# IPv6 -- only required when forwarding is enabled or
# the ipv6 configuration is disabled, e.g.:
#default fe80::201:1ff:fe01:254 - br0
- Network interface definition for the 'xen1' virtual machine:
/etc/xen/vm/xen1:
[...]
vif=[ 'mac=00:16:3e:10:10:10,script=vif-bridge,bridge=br0', ]
#vif=[ 'mac=00:16:3e:10:10:10,script=vif-bridge,bridge=br0,vifname=xen1.0', ]
[...]
Using 'vifname=xen1.0' causes to use of a static interface name and is
completely optional in bridged mode. It allows to filter the traffic
based on the interface name in firewall rules (see the 'physdev' match
extension description in the iptables(8) man page, for example:
'-m physdev --physdev-in xen1.0').
When the vifname is not specified, xen will use the 'vifX.Y' scheme
for the virtual machine network interfaces, that are different with
every start of the virtual machine.
No network configuration (ifcfg-xen1.0) for the virtual machine
interface is required on the host machine, because they are just
bridge port connectors same as the physical (eth0) interface.
The virtual machine network interfaces are added to the bridge
as ports by the vif-bridge script at startup time of the virtual
machine. The vif-bridge script is enabled by default in xen.
1.2) Host-Only Bridge
Host-Only networking can be realized using a bridge without adding any
physical ethernet network interface as bridge port.
+---------------------+
| HOST / dom0 | {other hosts}
| | . .
| <br0> <eth0>-[X]-------+ . .
| | | | | |
| +-[X]---...---[ ]-+ | +-[X]-[X]-...-[X]-+
| | bridge switch | | | physical |
| +-[X]---...---[X]-+ | | network switch |
| | | | +-[ ]-[ ]-...-[ ]-+
| <vif1.0> <vifN.0> |
| | | |
| +-[X]--+ +--[X]-+ |
| | dom1 | | domN | |
| +------+ +------+ |
+---------------------+
Linux supports multiple bridge interfaces.
For example br0 can be configured with a physical network interface as
port to provide access to a the physical network and br1 without any
physical interface port used as the host-only bridge.
The host-only bridge as well as the interfaces inside of the virtual
machines are usually configured using IP addresses from a private
network (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 as IPv4 networks).
It is possible, but not required to configure any private networks in
IPv6. The virtual machines and the host can communicate using so-called
'link-local' IPv6 addresses, that are auto-configured by default.
In case there are multiple networks required (e.g. there is more than
one "private" host-only network and the networks have to be connected
together), it is possible to use addresses e.g. from the fc00::/7 IPv6
prefix as specified in RFC 4193. Note, that these addresses are not
expected to be routable on the global Internet.
The host-only network services as e.g. dhcp- and dns-server have to be
bound to the host-only bridge interface and/or the IP addresses on the
bridge interfaces on the host system.
Alternatively, these services can be provided by one of the virtual
machines.
Example - host-only bridge br1, additionally to the 'br0' bridge
with physical network interface from previous chapter:
- Network configuration of host-only bridge in the host system:
/etc/sysconfig/network/ifcfg-br1:
STARTMODE='auto'
BOOTPROTO='static'
IPADDR='192.168.1.1/24'
BRIDGE='yes'
BRIDGE_PORTS=''
# ^^
# ||
# empty 'host-only' bridge
#
BRIDGE_STP='off'
BRIDGE_FORWARDDELAY='0'
- Network interface definition for the 'xen1' virtual machine:
/etc/xen/vm/xen1:
[...]
#vif=[ 'mac=00:16:3e:10:10:10,script=vif-bridge,bridge=br0', ]
vif=[ 'mac=00:16:3e:11:11:11,script=vif-bridge,bridge=br1', ]
## or also with static vifnames:
#vif=[ 'mac=00:16:3e:10:10:10,script=vif-bridge,bridge=br0,vifname=xen1.0', ]
#vif=[ 'mac=00:16:3e:11:11:11,script=vif-bridge,bridge=br1,vifname=xen1.1', ]
[...]
2) Routed setup
Routed networking basically means, there is no bridge involved and the
traffic (at least between the physical network and the virtual machines)
is routed instead.
Routed setup is usually required on machines in the internet, because
most ISPs will not allow any traffic with a different MAC address than
the address of the physical network interface and block when a bridge
is configured on the physical interface.
Further, it has to be used also in special cases, for example, when the
root file system is installed on iSCSI / NFS. It is not supported to add
any interface serving a root file system to a bridge, because it would
break the iSCSI / NFS connections serving the root file system.
The physical network interfaces used for root file systems are configured
already in the initrd and has to stay configured all the time.
In routed setup, the host machine is configured as router with enabled
IP forwarding:
* To enable IPv4 forwarding:
- set IP_FORWARD=yes in /etc/sysconfig/sysctl
- set FW_ROUTE=yes in /etc/sysconfig/SuSEfirewall2
* To enable IPv6 forwarding:
- set IPV6_FORWARD="yes" in /etc/sysconfig/sysctl
All further configuration depends on the IP address space availability,
the configuration on the physical network (router) as well as on the
requirements for the virtual machines.
It is possible and sometimes useful to use host-only bridges to group
virtual machines. For example, each host-only bridge represents one
network or group (one switch) and the host system is routing between
these networks and the physical network. The communication between the
virtual machines inside of one group is bridged then.
It is also often handy to keep the bridge connected to a physical network
or a host-only bridge e.g. for installation purposes.
Further, host-only bridges are also usable as workaround to handle dynamic
interface names (vifX.Y) used in XEN by default.
Note: Older XEN versions may not support the 'vifname' keyword allowing
to specify a custom and static interface name for full-virtualized guests.
Two major cases have to be considered:
- There is only one shared network that has to be used on both, the
physical interface on the host system and on the virtual machines.
- There are multiple subnets. One is assigned to the physical interface,
further virtual machine networks are routed through this network.
2.1) Routed setup with one shared network example:
When there is only one network (/64 network in case of IPv6) available
for the host system and the virtual machines, the setup requires the
use of IPv4 Proxy ARP and IPv6 Proxy NDP.
IPv4:
Network Address : 192.0.2.240/28 (255.255.255.240)
Gateway/Router : 192.0.2.254
Host System IP : 192.0.2.253
Virtual Machine IPs : 192.0.2.241 .. 192.0.2.252
IPv6:
Network Prefix : 2001:db8:abba::/64
Gateway/Router : fe80::201:1ff:fe01:254
Host System IP : 2001:db8:abba::1/64
Virtual Machine IPs : 2001:db8:abba::1001 .. 1ffe (/116)
We can't subnet a IPv6 /64 network any more and have to proxy each IPv6
address separately. Routes can also be specified using normal CIDR
notation and don't need to be routed one by one.
We route the IPv6 and also the IPv4 addresses separately, because at
least each IPv4 address is used by a different virtual machine.
To enable IPv4/IPv6 forwarding and proxy_arp/proxy_ndp set:
- IP_FORWARD=yes and IPV6_FORWARD=yes in /etc/sysconfig/sysctl
- When IPv6 is compiled as module, append "ipv6" at the end of the
INITRD_MODULES variable in /etc/sysconfig/kernel, so the module
is loaded early enough the boot.sysctl execution.
- set/add following settings in /etc/sysconfig/network/ifsysctl:
# enable ipv4 proxy arp and forwarding:
net.ipv4.conf.all.proxy_arp = 1
net.ipv4.conf.all.forwarding = 1
# enable ipv6 proxy ndp and forwarding:
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.all.proxy_ndp = 1
and apply them:
mkinitrd
update-bootloader --refresh
/etc/init.d/boot.sysctl start
/etc/init.d/boot.ipconfig start
Note, that the ifsysctl will be applied in every ifup call, that is,
on every virtual machine start (executed by vif-route-ifup).
See also special proxy-arp case in the SuSEfirewall2 section bellow.
- Network configuration in the host system:
/etc/sysconfig/network/ifcfg-eth0:
BOOTPROTO='static'
STARTMODE='onboot'
IPADDR='192.0.2.253/28'
IPADDR_1='2001:db8:abba::1/64'
/etc/sysconfig/network/ifcfg-xen1.0
NAME='XEN vm 1 virtual interface 0'
BOOTPROTO='static'
STARTMODE='hotplug'
# ^^^^^^^
# to avoid attempts to start/wait for it at boot time
/etc/sysconfig/network/routes (or ifroute-eth0, ...):
# IPv4
# - first virtual machine:
192.0.2.241 - - xen1.0
# - default route:
default 192.0.2.254 - eth0
# IPv6
# - first virtual machine:
2001:db8:abba::1001 fe80::216:3eff:fe10:1010 - xen1.0
# default -- required because forwarding is enabled
default fe80::201:1ff:fe01:254 - eth0
Finally, it is required to publish the addresses between the
interfaces - this can be done in the ifndp-proxy file
/etc/sysconfig/network/ifndp-proxy:
# <address> <address interface> <proxy interface>
192.0.2.253 eth0 xen1.0
2001:db8:abba::1 eth0 xen1.0
192.0.2.241 xen1.0 eth0
2001:db8:abba::1001 xen1.0 eth0
See also description and examples in the ifndp-proxy(5) manual page.
- Network configuration inside of the 1st virtual machine:
/etc/sysconfig/network/ifcfg-eth0:
NAME='Virtual Ethernet Card 0'
BOOTPROTO='static'
STARTMODE='auto'
IPADDR='192.0.2.241/32'
IPADDR_1='2001:db8:abba::1001/64'
/etc/sysconfig/network/routes (or ifroute-eth0)
# IPv4
192.0.2.253 - - eth0
default 192.0.2.253 - eth0
# IPv6
#default fe80::fcff:ffff:feff:ffff - eth0
default 2001:db8:abba::1 - eth0
- Network interface definition for the 'xen1' virtual machine:
In the virtual machine config, we set script to vif-route-ifup
and the vifname to some static interface name, e.g. xen1.0:
/etc/xen/vm/xen1:
[...]
vif=[ 'mac=00:16:3e:10:10:10,script=vif-route-ifup,vifname=xen1.0', ]
[...]
The use of the vifname keyword is required to get a static interface
name instead of the dynamic vifX.Y names used by XEN by default, so
we have a static name for the above sysconfig configuration files.
The vif-route-ifup script is a stripped down vif-route, calling the
"ifup ${vif}" and "do_without_error ifdown ${vif}" command only.
It is provided together with another xen scripts on SLE11-SP1.
The fe80::216:3eff:fe10:1010 address is derived from the MAC address,
that has been set in the vif= line of the virtual machine config file
(00:16:3e:10:10:10). You can either calculate it yourself or start the
machine, attach a console, and take a look to the "ip addr show" output
inside of the machine, e.g.:
[...]
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> [...]
link/ether 00:16:3e:10:10:10 brd ff:ff:ff:ff:ff:ff
inet6 fe80::216:3eff:fe10:1010/64 scope link
[...]
For more informations, search for "EUI-64 Identifiers" - there are
several pages with nice calculation instructions available on the
internet.
- SuSEfirewall2 settings on the host system:
SuSEfirewall2 disables proxy-arp every time it runs and there is
currently no variable to change this, except of a custom hook
like /etc/sysconfig/scripts/SuSEfirewall2-custom:
fw_custom_after_antispoofing() {
/etc/sysconfig/network/scripts/ifup-sysctl all
true
}
The name of the script has to be specified in the config:
FW_CUSTOMRULES="/etc/sysconfig/scripts/SuSEfirewall2-custom"
Further, you may need to allow forwarding from the virtual
machine to another networks, like:
FW_FORWARD="192.0.2.241,0/0"
for this examples. Don't forget to set also FW_ROUTE=yes.
2.2) Routed setup with multiple networks
In this scenario, we don't need to set up any proxy-arp/proxy-ndp rules.
Except of this, the setup is equivalent to the setup of a shared network.
To make it more interesting, we set up a host-only bridge for each
virtual machine group, so the host system can route complete networks
to these groups.
Each virtual machine can have an interface in one or more of these groups
(bridges) when needed.
This allows also to configure and use e.g. radvd, dhcp/dhcp6 servers
started on the bridge interfaces, so the virtual machines can assign
the IPv4/IPv6 address, routing, and further configuration them self.
- Network at the physical interface:
IPv4:
Network Address : 192.0.2.240/28 (255.255.255.240)
Gateway/Router : 192.0.2.254
Host System IP : 192.0.2.253
IPv6:
Gateway/Router : fe80::201:1ff:fe01:254
Interface Prefix : 2001:db8:abba:2000::/64
Host System IP : 2001:db8:abba:2000::1/64
Prefix routed to host : 2001:db8:abba:2000::/60
(networks 2001:0db8:abba:2000::/64 .. 2001:0db8:abba:200f::/64
with the 2001:0db8:abba:2000::/64 network for the host system)
- Network for 1. virtual machine group / bridge:
IPv4:
Network Address : 192.0.2.0/28 (255.255.255.240)
Host System IP : 192.0.2.14
Virtual Machine IPs : 192.0.2.1 .. 192.0.2.13
IPv6:
Network Prefix : 2001:db8:abba:2001::/64
- Network for 2. virtual machine group / bridge:
IPv4:
Network Address : 192.0.2.16/28 (255.255.255.240)
Host System IP : 192.0.2.30
Virtual Machine IPs : 192.0.2.17 .. 192.0.2.29
IPv6:
Network Prefix : 2001:db8:abba:2002::/64
To enable IPv4/IPv6 forwarding, set:
- IP_FORWARD=yes and IPV6_FORWARD=yes in /etc/sysconfig/sysctl
- When IPv6 is compiled as module, append "ipv6" at the end of the
INITRD_MODULES variable in /etc/sysconfig/kernel, so the module
is loaded early enough the boot.sysctl execution.
- set/add following settings in /etc/sysconfig/network/ifsysctl:
# enable ipv4 forwarding:
net.ipv4.conf.all.forwarding = 1
# enable ipv6 forwarding:
net.ipv6.conf.all.forwarding = 1
and apply them:
mkinitrd
update-bootloader --refresh
/etc/init.d/boot.sysctl start
/etc/init.d/boot.ipconfig start
Note, that the ifsysctl will be applied in every ifup call, that is,
on every virtual machine start.
- Network configuration in the host system:
/etc/sysconfig/network/ifcfg-eth0:
STARTMODE='auto'
BOOTPROTO='static'
IPADDR='192.0.2.253/28'
IPADDR_1='2001:db8:abba:2000::1/64'
/etc/sysconfig/network/ifcfg-br1:
STARTMODE='auto'
BOOTPROTO='static'
IPADDR='192.0.2.14/28'
BRIDGE='yes'
BRIDGE_PORTS=''
BRIDGE_STP='off'
BRIDGE_FORWARDDELAY='0'
/etc/sysconfig/network/ifcfg-br2:
STARTMODE='auto'
BOOTPROTO='static'
IPADDR='192.0.2.30/28'
BRIDGE='yes'
BRIDGE_PORTS=''
BRIDGE_STP='off'
BRIDGE_FORWARDDELAY='0'
/etc/sysconfig/network/routes (or ifroute-eth0, br1, br2)
# IPv4
default 192.0.2.254 - eth0
# IPv6
# - avoid that traffic to our network goes out
# via default route back to default gateway
2001:db8:abba:2000::/60 - - - unreachable
2001:db8:abba:2001::/64 - - br1
2001:db8:abba:2002::/64 - - br2
# - required, because forwarding is enabled
default fe80::201:1ff:fe01:254 - eth0
Note, we didn't set any IPv6 address on the bridges as we did for IPv4;
it is possible to assign them, but not required.
- Router Advertisement setup on the host system
The radvd daemon allows the virtual machines to assign IPv6 addresses
and routing them self. Enable it "insserv /etc/init.d/radvd" and start.
/etc/radvd.conf:
interface br1
{
IgnoreIfMissing on;
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
# enable stateful autoconfig via dhcpv6:
#AdvManagedFlag on;
#AdvOtherConfigFlag on;
prefix 2001:db8:abba:2001::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr off;
};
#RDNSS 2001:db8:abba:2000:1
#{
#};
};
interface br2
{
IgnoreIfMissing on;
AdvSendAdvert on;
MinRtrAdvInterval 3;
MaxRtrAdvInterval 10;
# to enable stateful autoconfig via dhcpv6:
#AdvManagedFlag on;
#AdvOtherConfigFlag on;
prefix 2001:db8:abba:2002::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr off;
};
#RDNSS 2001:db8:abba:2000:1
#{
#};
};
The RDNSS statement does not work, so when you want to provide DNS
informations to the virtual machines, install and configure dhcpv6
server -- additionally to radvd -- and enable the AdvManagedFlag.
- Network configuration inside of the xen1 virtual machine:
/etc/sysconfig/network/ifcfg-eth0:
NAME='Virtual Ethernet Card 0'
BOOTPROTO='static'
STARTMODE='auto'
IPADDR='192.0.2.1/28'
/etc/sysconfig/network/routes (or ifroute-eth0)
# IPv4
192.0.2.14 - - eth0
default 192.0.2.14 - eth0
Note, the IPv6 address and default route are assigned via radvd.
- Network interface definition for the 'xen1' virtual machine:
In the virtual machine config, we set script to vif-bridge:
/etc/xen/vm/xen1:
[...]
vif=[ 'mac=00:16:3e:10:10:10,bridge=br1,vifname=xen1.0', ]
[...]
The xen and network configuration inside of another virtual machines
is equivalent to the above one.
3) Routed setup with NAT
A NAT setup is basically a routed setup. The virtual machines are using
private IP addresses and the NAT rules map the available external IPv4
addresses (official IPs) per IP or even on a port basis to the virtual
machines.
The setup is similar to 2.1 (without proxy-arp/ndp) or 2.2 using private
addresses for the virtual machines and map using iptables NAT/masquerade
rules. See /etc/sysconfig/SuSEfirewall2 variables:
FW_REDIRECT,
FW_MASQ_NETS, FW_MASQ_DEV,
FW_FORWARD, FW_FORWARD_REJECT,
FW_FORWARD_MASQ, FW_FORWARD_MASQ,
and the examples provided by the SuSEfirewall2. Note, that there is no
'nat' in IPv6 ip6tables.
Routes with source IP as it is used in xen's vif-nat script:
$ ip route add 192.168.1.10 dev xen1.0 src 192.168.1.1
can be written in /etc/sysconfig/network/routes as follows:
192.168.1.10 - - xen1.0 src 192.168.1.1
ACC SHELL 2018