1 How to Use Open vSwitch with Docker
2 ====================================
4 This document describes how to use Open vSwitch with Docker 1.9.0 or
5 later. This document assumes that you installed Open vSwitch by following
6 [INSTALL.md] or by using the distribution packages such as .deb or .rpm.
7 Consult www.docker.com for instructions on how to install Docker.
9 Docker 1.9.0 comes with support for multi-host networking. Integration
10 of Docker networking and Open vSwitch can be achieved via Open vSwitch
11 virtual network (OVN).
17 For multi-host networking with OVN and Docker, Docker has to be started
18 with a destributed key-value store. For e.g., if you decide to use consul
19 as your distributed key-value store, and your host IP address is $HOST_IP,
20 start your Docker daemon with:
23 docker daemon --cluster-store=consul://127.0.0.1:8500 \
24 --cluster-advertise=$HOST_IP:0
27 OVN provides network virtualization to containers. OVN's integration with
28 Docker currently works in two modes - the "underlay" mode or the "overlay"
31 In the "underlay" mode, OVN requires a OpenStack setup to provide container
32 networking. In this mode, one can create logical networks and can have
33 containers running inside VMs, standalone VMs (without having any containers
34 running inside them) and physical machines connected to the same logical
35 network. This is a multi-tenant, multi-host solution.
37 In the "overlay" mode, OVN can create a logical network amongst containers
38 running on multiple hosts. This is a single-tenant (extendable to
39 multi-tenants depending on the security characteristics of the workloads),
40 multi-host solution. In this mode, you do not need a pre-created OpenStack
43 For both the modes to work, a user has to install and start Open vSwitch in
44 each VM/host that he plans to run his containers.
50 OVN in "overlay" mode needs a minimum Open vSwitch version of 2.5.
52 * Start the central components.
54 OVN architecture has a central component which stores your networking intent
55 in a database. On one of your machines, with an IP Address of $CENTRAL_IP,
56 where you have installed and started Open vSwitch, you will need to start some
59 Begin by making ovsdb-server listen on a TCP port by running:
62 ovs-appctl -t ovsdb-server ovsdb-server/add-remote ptcp:6640
65 Start ovn-northd daemon. This daemon translates networking intent from Docker
66 stored in the OVN_Northbound database to logical flows in OVN_Southbound
70 /usr/share/openvswitch/scripts/ovn-ctl start_northd
75 On each host, where you plan to spawn your containers, you will need to
76 run the following command once. (You need to run it again if your OVS database
77 gets cleared. It is harmless to run it again in any case.)
79 $LOCAL_IP in the below command is the IP address via which other hosts
80 can reach this host. This acts as your local tunnel endpoint.
82 $ENCAP_TYPE is the type of tunnel that you would like to use for overlay
83 networking. The options are "geneve" or "stt". (Please note that your
84 kernel should have support for your chosen $ENCAP_TYPE. Both geneve
85 and stt are part of the Open vSwitch kernel module that is compiled from this
86 repo. If you use the Open vSwitch kernel module from upstream Linux,
87 you will need a minumum kernel version of 3.18 for geneve. There is no stt
88 support in upstream Linux. You can verify whether you have the support in your
89 kernel by doing a "lsmod | grep $ENCAP_TYPE".)
92 ovs-vsctl set Open_vSwitch . external_ids:ovn-remote="tcp:$CENTRAL_IP:6640" \
93 external_ids:ovn-encap-ip=$LOCAL_IP external_ids:ovn-encap-type="$ENCAP_TYPE"
96 And finally, start the ovn-controller. (You need to run the below command
100 /usr/share/openvswitch/scripts/ovn-ctl start_controller
103 * Start the Open vSwitch network driver.
105 By default Docker uses Linux bridge for networking. But it has support
106 for external drivers. To use Open vSwitch instead of the Linux bridge,
107 you will need to start the Open vSwitch driver.
109 The Open vSwitch driver uses the Python's flask module to listen to
110 Docker's networking api calls. So, if your host does not have Python's
111 flask module, install it with:
118 Start the Open vSwitch driver on every host where you plan to create your
122 ovn-docker-overlay-driver --detach
125 Docker has inbuilt primitives that closely match OVN's logical switches
126 and logical port concepts. Please consult Docker's documentation for
127 all the possible commands. Here are some examples.
129 * Create your logical switch.
131 To create a logical switch with name 'foo', on subnet '192.168.1.0/24' run:
134 NID=`docker network create -d openvswitch --subnet=192.168.1.0/24 foo`
137 * List your logical switches.
143 You can also look at this logical switch in OVN's northbound database by
144 running the following command.
147 ovn-nbctl --db=tcp:$CENTRAL_IP:6640 lswitch-list
150 * Docker creates your logical port and attaches it to the logical network
153 For e.g., to attach a logical port to network 'foo' inside cotainer busybox,
157 docker run -itd --net=foo --name=busybox busybox
160 * List all your logical ports.
162 Docker currently does not have a CLI command to list all your logical ports.
163 But you can look at them in the OVN database, by running:
166 ovn-nbctl --db=tcp:$CENTRAL_IP:6640 lport-list $NID
169 * You can also create a logical port and attach it to a running container.
172 docker network create -d openvswitch --subnet=192.168.2.0/24 bar
173 docker network connect bar busybox
176 You can delete your logical port and detach it from a running container by
180 docker network disconnect bar busybox
183 * You can delete your logical switch by running:
186 docker network rm bar
193 This mode requires that you have a OpenStack setup pre-installed with OVN
194 providing the underlay networking.
198 A OpenStack tenant creates a VM with a single network interface (or multiple)
199 that belongs to management logical networks. The tenant needs to fetch the
200 port-id associated with the interface via which he plans to send the container
201 traffic inside the spawned VM. This can be obtained by running the
202 below command to fetch the 'id' associated with the VM.
211 neutron port-list --device_id=$id
214 Inside the VM, download the OpenStack RC file that contains the tenant
215 information (henceforth referred to as 'openrc.sh'). Edit the file and add the
216 previously obtained port-id information to the file by appending the following
217 line: export OS_VIF_ID=$port_id. After this edit, the file will look something
222 export OS_AUTH_URL=http://10.33.75.122:5000/v2.0
223 export OS_TENANT_ID=fab106b215d943c3bad519492278443d
224 export OS_TENANT_NAME="demo"
225 export OS_USERNAME="demo"
226 export OS_VIF_ID=e798c371-85f4-4f2d-ad65-d09dd1d3c1c9
229 * Create the Open vSwitch bridge.
231 If your VM has one ethernet interface (e.g.: 'eth0'), you will need to add
232 that device as a port to an Open vSwitch bridge 'breth0' and move its IP
233 address and route related information to that bridge. (If it has multiple
234 network interfaces, you will need to create and attach an Open vSwitch bridge
235 for the interface via which you plan to send your container traffic.)
237 If you use DHCP to obtain an IP address, then you should kill the DHCP client
238 that was listening on the physical Ethernet interface (e.g. eth0) and start
239 one listening on the Open vSwitch bridge (e.g. breth0).
241 Depending on your VM, you can make the above step persistent across reboots.
242 For e.g.:, if your VM is Debian/Ubuntu, you can read
243 [openvswitch-switch.README.Debian]. If your VM is RHEL based, you can read
247 * Start the Open vSwitch network driver.
249 The Open vSwitch driver uses the Python's flask module to listen to
250 Docker's networking api calls. The driver also uses OpenStack's
251 python-neutronclient libraries. So, if your host does not have Python's
252 flask module or python-neutronclient install them with:
256 pip install python-neutronclient
260 Source the openrc file. e.g.:
265 Start the network driver and provide your OpenStack tenant password
269 ovn-docker-underlay-driver --bridge breth0 --detach
272 From here-on you can use the same Docker commands as described in the
273 section 'The "overlay" mode'.
275 Please read 'man ovn-architecture' to understand OVN's architecture in
278 [INSTALL.md]: INSTALL.md
279 [openvswitch-switch.README.Debian]: debian/openvswitch-switch.README.Debian
280 [README.RHEL]: rhel/README.RHEL