1 <?xml version="1.0" encoding="utf-8"?>
2 <manpage program="ovn-architecture" section="7" title="OVN Architecture">
4 <p>ovn-architecture -- Open Virtual Network architecture</p>
9 OVN, the Open Virtual Network, is a system to support virtual network
10 abstraction. OVN complements the existing capabilities of OVS to add
11 native support for virtual network abstractions, such as virtual L2 and L3
12 overlays and security groups. Services such as DHCP are also desirable
13 features. Just like OVS, OVN's design goal is to have a production-quality
14 implementation that can operate at significant scale.
18 An OVN deployment consists of several components:
24 A <dfn>Cloud Management System</dfn> (<dfn>CMS</dfn>), which is
25 OVN's ultimate client (via its users and administrators). OVN
26 integration requires installing a CMS-specific plugin and
27 related software (see below). OVN initially targets OpenStack
32 We generally speak of ``the'' CMS, but one can imagine scenarios in
33 which multiple CMSes manage different parts of an OVN deployment.
38 An OVN Database physical or virtual node (or, eventually, cluster)
39 installed in a central location.
43 One or more (usually many) <dfn>hypervisors</dfn>. Hypervisors must run
44 Open vSwitch and implement the interface described in
45 <code>IntegrationGuide.md</code> in the OVS source tree. Any hypervisor
46 platform supported by Open vSwitch is acceptable.
51 Zero or more <dfn>gateways</dfn>. A gateway extends a tunnel-based
52 logical network into a physical network by bidirectionally forwarding
53 packets between tunnels and a physical Ethernet port. This allows
54 non-virtualized machines to participate in logical networks. A gateway
55 may be a physical host, a virtual machine, or an ASIC-based hardware
56 switch that supports the <code>vtep</code>(5) schema. (Support for the
57 latter will come later in OVN implementation.)
61 Hypervisors and gateways are together called <dfn>transport node</dfn>
62 or <dfn>chassis</dfn>.
68 The diagram below shows how the major components of OVN and related
69 software interact. Starting at the top of the diagram, we have:
74 The Cloud Management System, as defined above.
79 The <dfn>OVN/CMS Plugin</dfn> is the component of the CMS that
80 interfaces to OVN. In OpenStack, this is a Neutron plugin.
81 The plugin's main purpose is to translate the CMS's notion of logical
82 network configuration, stored in the CMS's configuration database in a
83 CMS-specific format, into an intermediate representation understood by
88 This component is necessarily CMS-specific, so a new plugin needs to be
89 developed for each CMS that is integrated with OVN. All of the
90 components below this one in the diagram are CMS-independent.
96 The <dfn>OVN Northbound Database</dfn> receives the intermediate
97 representation of logical network configuration passed down by the
98 OVN/CMS Plugin. The database schema is meant to be ``impedance
99 matched'' with the concepts used in a CMS, so that it directly supports
100 notions of logical switches, routers, ACLs, and so on. See
101 <code>ovs-nb</code>(5) for details.
105 The OVN Northbound Database has only two clients: the OVN/CMS Plugin
106 above it and <code>ovn-nbd</code> below it.
111 <code>ovn-nbd</code>(8) connects to the OVN Northbound Database above it
112 and the OVN Database below it. It translates the logical network
113 configuration in terms of conventional network concepts, taken from the
114 OVN Northbound Database, into logical datapath flows in the OVN Database
120 The <dfn>OVN Database</dfn> is the center of the system. Its clients
121 are <code>ovn-nbd</code>(8) above it and <code>ovn-controller</code>(8)
122 on every transport node below it.
126 The OVN Database contains three kinds of data: <dfn>Physical
127 Network</dfn> (PN) tables that specify how to reach hypervisor and
128 other nodes, <dfn>Logical Network</dfn> (LN) tables that describe the
129 logical network in terms of ``logical datapath flows,'' and
130 <dfn>Binding</dfn> tables that link logical network components'
131 locations to the physical network. The hypervisors populate the PN and
132 Binding tables, whereas <code>ovn-nbd</code>(8) populates the LN
137 OVN Database performance must scale with the number of transport nodes.
138 This will likely require some work on <code>ovsdb-server</code>(1) as
139 we encounter bottlenecks. Clustering for availability may be needed.
145 The remaining components are replicated onto each hypervisor:
150 <code>ovn-controller</code>(8) is OVN's agent on each hypervisor and
151 software gateway. Northbound, it connects to the OVN Database to learn
152 about OVN configuration and status and to populate the PN table and the
153 <code>Chassis</code> column in <code>Bindings</code> table with the
154 hypervisor's status. Southbound, it connects to
155 <code>ovs-vswitchd</code>(8) as an OpenFlow controller, for control over
156 network traffic, and to the local <code>ovsdb-server</code>(1) to allow
157 it to monitor and control Open vSwitch configuration.
161 <code>ovs-vswitchd</code>(8) and <code>ovsdb-server</code>(1) are
162 conventional components of Open vSwitch.
170 +-----------|-----------+
175 | OVN Northbound DB |
180 +-----------|-----------+
188 +------------------+------------------+
191 +---------------|---------------+ . +---------------|---------------+
193 | ovn-controller | . | ovn-controller |
196 | ovs-vswitchd ovsdb-server | | ovs-vswitchd ovsdb-server |
198 +-------------------------------+ +-------------------------------+
201 <h2>Chassis Setup</h2>
204 Each chassis in an OVN deployment must be configured with an Open vSwitch
205 bridge dedicated for OVN's use, called the <dfn>integration bridge</dfn>.
206 System startup scripts create this bridge prior to starting
207 <code>ovn-controller</code>. The ports on the integration bridge include:
212 On any chassis, tunnel ports that OVN uses to maintain logical network
213 connectivity. <code>ovn-controller</code> adds, updates, and removes
218 On a hypervisor, any VIFs that are to be attached to logical networks.
219 The hypervisor itself, or the integration between Open vSwitch and the
220 hypervisor (described in <code>IntegrationGuide.md</code>) takes care of
221 this. (This is not part of OVN or new to OVN; this is pre-existing
222 integration work that has already been done on hypervisors that support
227 On a gateway, the physical port used for logical network connectivity.
228 System startup scripts add this port to the bridge prior to starting
229 <code>ovn-controller</code>. This can be a patch port to another bridge,
230 instead of a physical port, in more sophisticated setups.
235 Other ports should not be attached to the integration bridge. In
236 particular, physical ports attached to the underlay network (as opposed to
237 gateway ports, which are physical ports attached to logical networks) must
238 not be attached to the integration bridge. Underlay physical ports should
239 instead be attached to a separate Open vSwitch bridge (they need not be
240 attached to any bridge at all, in fact).
244 The integration bridge must be configured with failure mode ``secure'' to
245 avoid switching packets between isolated logical networks before
246 <code>ovn-controller</code> starts up. See <code>Controller Failure
247 Settings</code> in <code>ovs-vsctl</code>(8) for more information.
251 The customary name for the integration bridge is <code>br-int</code>, but
252 another name may be used.
255 <h2>Life Cycle of a VIF</h2>
258 Tables and their schemas presented in isolation are difficult to
259 understand. Here's an example.
263 A VIF on a hypervisor is a virtual network interface attached either
264 to a VM or a container running directly on that hypervisor (This is
265 different from the interface of a container running inside a VM).
269 The steps in this example refer often to details of the OVN and OVN
270 Northbound database schemas. Please see <code>ovn</code>(5) and
271 <code>ovn-nb</code>(5), respectively, for the full story on these
277 A VIF's life cycle begins when a CMS administrator creates a new VIF
278 using the CMS user interface or API and adds it to a switch (one
279 implemented by OVN as a logical switch). The CMS updates its own
280 configuration. This includes associating unique, persistent identifier
281 <var>vif-id</var> and Ethernet address <var>mac</var> with the VIF.
285 The CMS plugin updates the OVN Northbound database to include the new
286 VIF, by adding a row to the <code>Logical_Port</code> table. In the new
287 row, <code>name</code> is <var>vif-id</var>, <code>mac</code> is
288 <var>mac</var>, <code>switch</code> points to the OVN logical switch's
289 Logical_Switch record, and other columns are initialized appropriately.
293 <code>ovs-nbd</code> receives the OVN Northbound database update. In
294 turn, it makes the corresponding updates to the OVN database, by adding
295 rows to the OVN database <code>Pipeline</code> table to reflect the new
296 port, e.g. add a flow to recognize that packets destined to the new
297 port's MAC address should be delivered to it, and update the flow that
298 delivers broadcast and multicast packets to include the new port. It
299 also creates a record in the <code>Bindings</code> table and populates
300 all its columns except the column that identifies the
301 <code>chassis</code>.
305 On every hypervisor, <code>ovn-controller</code> receives the
306 <code>Pipeline</code> table updates that <code>ovs-nbd</code> made in the
307 previous step. As long as the VM that owns the VIF is powered off,
308 <code>ovn-controller</code> cannot do much; it cannot, for example,
309 arrange to send packets to or receive packets from the VIF, because the
310 VIF does not actually exist anywhere.
314 Eventually, a user powers on the VM that owns the VIF. On the hypervisor
315 where the VM is powered on, the integration between the hypervisor and
316 Open vSwitch (described in <code>IntegrationGuide.md</code>) adds the VIF
317 to the OVN integration bridge and stores <var>vif-id</var> in
318 <code>external-ids</code>:<code>iface-id</code> to indicate that the
319 interface is an instantiation of the new VIF. (None of this code is new
320 in OVN; this is pre-existing integration work that has already been done
321 on hypervisors that support OVS.)
325 On the hypervisor where the VM is powered on, <code>ovn-controller</code>
326 notices <code>external-ids</code>:<code>iface-id</code> in the new
327 Interface. In response, it updates the local hypervisor's OpenFlow
328 tables so that packets to and from the VIF are properly handled.
329 Afterward, in the OVN DB, it updates the <code>Bindings</code> table's
330 <code>chassis</code> column for the row that links the logical port
331 from <code>external-ids</code>:<code>iface-id</code> to the hypervisor.
335 Some CMS systems, including OpenStack, fully start a VM only when its
336 networking is ready. To support this, <code>ovn-nbd</code> notices the
337 <code>chassis</code> column updated for the row in <code>Bindings</code>
338 table and pushes this upward by updating the <ref column="up"
339 table="Logical_Port" db="OVN_NB"/> column in the OVN Northbound
340 database's <ref table="Logical_Port" db="OVN_NB"/> table to indicate
341 that the VIF is now up. The CMS, if it uses this feature, can then
342 react by allowing the VM's execution to proceed.
346 On every hypervisor but the one where the VIF resides,
347 <code>ovn-controller</code> notices the completely populated row in the
348 <code>Bindings</code> table. This provides <code>ovn-controller</code>
349 the physical location of the logical port, so each instance updates the
350 OpenFlow tables of its switch (based on logical datapath flows in the OVN
351 DB <code>Pipeline</code> table) so that packets to and from the VIF can
352 be properly handled via tunnels.
356 Eventually, a user powers off the VM that owns the VIF. On the
357 hypervisor where the VM was powered on, the VIF is deleted from the OVN
362 On the hypervisor where the VM was powered on,
363 <code>ovn-controller</code> notices that the VIF was deleted. In
364 response, it removes the <code>Chassis</code> column content in the
365 <code>Bindings</code> table for the logical port.
369 On every hypervisor, <code>ovn-controller</code> notices the empty
370 <code>Chassis</code> column in the <code>Bindings</code> table's row
371 for the logical port. This means that <code>ovn-controller</code> no
372 longer knows the physical location of the logical port, so each instance
373 updates its OpenFlow table to reflect that.
377 Eventually, when the VIF (or its entire VM) is no longer needed by
378 anyone, an administrator deletes the VIF using the CMS user interface or
379 API. The CMS updates its own configuration.
383 The CMS plugin removes the VIF from the OVN Northbound database,
384 by deleting its row in the <code>Logical_Port</code> table.
388 <code>ovs-nbd</code> receives the OVN Northbound update and in turn
389 updates the OVN database accordingly, by removing or updating the
390 rows from the OVN database <code>Pipeline</code> table and
391 <code>Bindings</code> table that were related to the now-destroyed VIF.
395 On every hypervisor, <code>ovn-controller</code> receives the
396 <code>Pipeline</code> table updates that <code>ovs-nbd</code> made in the
397 previous step. <code>ovn-controller</code> updates OpenFlow tables to
398 reflect the update, although there may not be much to do, since the VIF
399 had already become unreachable when it was removed from the
400 <code>Bindings</code> table in a previous step.
404 <h2>Life Cycle of a container interface inside a VM</h2>
407 OVN provides virtual network abstractions by converting information
408 written in OVN_NB database to OpenFlow flows in each hypervisor. Secure
409 virtual networking for multi-tenants can only be provided if OVN controller
410 is the only entity that can modify flows in Open vSwitch. When the
411 Open vSwitch integration bridge resides in the hypervisor, it is a
412 fair assumption to make that tenant workloads running inside VMs cannot
413 make any changes to Open vSwitch flows.
417 If the infrastructure provider trusts the applications inside the
418 containers not to break out and modify the Open vSwitch flows, then
419 containers can be run in hypervisors. This is also the case when
420 containers are run inside the VMs and Open vSwitch integration bridge
421 with flows added by OVN controller resides in the same VM. For both
422 the above cases, the workflow is the same as explained with an example
423 in the previous section ("Life Cycle of a VIF").
427 This section talks about the life cycle of a container interface (CIF)
428 when containers are created in the VMs and the Open vSwitch integration
429 bridge resides inside the hypervisor. In this case, even if a container
430 application breaks out, other tenants are not affected because the
431 containers running inside the VMs cannot modify the flows in the
432 Open vSwitch integration bridge.
436 When multiple containers are created inside a VM, there are multiple
437 CIFs associated with them. The network traffic associated with these
438 CIFs need to reach the Open vSwitch integration bridge running in the
439 hypervisor for OVN to support virtual network abstractions. OVN should
440 also be able to distinguish network traffic coming from different CIFs.
441 There are two ways to distinguish network traffic of CIFs.
445 One way is to provide one VIF for every CIF (1:1 model). This means that
446 there could be a lot of network devices in the hypervisor. This would slow
447 down OVS because of all the additional CPU cycles needed for the management
448 of all the VIFs. It would also mean that the entity creating the
449 containers in a VM should also be able to create the corresponding VIFs in
454 The second way is to provide a single VIF for all the CIFs (1:many model).
455 OVN could then distinguish network traffic coming from different CIFs via
456 a tag written in every packet. OVN uses this mechanism and uses VLAN as
457 the tagging mechanism.
462 A CIF's life cycle begins when a container is spawned inside a VM by
463 the either the same CMS that created the VM or a tenant that owns that VM
464 or even a container Orchestration System that is different than the CMS
465 that initially created the VM. Whoever the entity is, it will need to
466 know the <var>vif-id</var> that is associated with the network interface
467 of the VM through which the container interface's network traffic is
468 expected to go through. The entity that creates the container interface
469 will also need to choose an unused VLAN inside that VM.
473 The container spawning entity (either directly or through the CMS that
474 manages the underlying infrastructure) updates the OVN Northbound
475 database to include the new CIF, by adding a row to the
476 <code>Logical_Port</code> table. In the new row, <code>name</code> is
477 any unique identifier, <code>parent_name</code> is the <var>vif-id</var>
478 of the VM through which the CIF's network traffic is expected to go
479 through and the <code>tag</code> is the VLAN tag that identifies the
480 network traffic of that CIF.
484 <code>ovn-nbd</code> receives the OVN Northbound database update. In
485 turn, it makes the corresponding updates to the OVN database, by adding
486 rows to the OVN database's <code>Pipeline</code> table to reflect the new
487 port and also by creating a new row in the <code>Bindings</code> table
488 and populating all its columns except the column that identifies the
489 <code>chassis</code>.
493 On every hypervisor, <code>ovn-controller</code> subscribes to the
494 changes in the <code>Bindings</code> table. When a new row is created
495 by <code>ovn-nbd</code> that includes a value in <code>parent_port</code>
496 column of <code>Bindings</code> table, the <code>ovn-controller</code>
497 in the hypervisor whose OVN integration bridge has that same value in
498 <var>vif-id</var> in <code>external-ids</code>:<code>iface-id</code>
499 updates the local hypervisor's OpenFlow tables so that packets to and
500 from the VIF with the particular VLAN <code>tag</code> are properly
501 handled. Afterward it updates the <code>chassis</code> column of
502 the <code>Bindings</code> to reflect the physical location.
506 One can only start the application inside the container after the
507 underlying network is ready. To support this, <code>ovn-nbd</code>
508 notices the updated <code>chassis</code> column in <code>Bindings</code>
509 table and updates the <ref column="up" table="Logical_Port"
510 db="OVN_NB"/> column in the OVN Northbound database's
511 <ref table="Logical_Port" db="OVN_NB"/> table to indicate that the
512 CIF is now up. The entity responsible to start the container application
513 queries this value and starts the application.
517 Eventually the entity that created and started the container, stops it.
518 The entity, through the CMS (or directly) deletes its row in the
519 <code>Logical_Port</code> table.
523 <code>ovn-nbd</code> receives the OVN Northbound update and in turn
524 updates the OVN database accordingly, by removing or updating the
525 rows from the OVN database <code>Pipeline</code> table that were related
526 to the now-destroyed CIF. It also deletes the row in the
527 <code>Bindings</code> table for that CIF.
531 On every hypervisor, <code>ovn-controller</code> receives the
532 <code>Pipeline</code> table updates that <code>ovn-nbd</code> made in the
533 previous step. <code>ovn-controller</code> updates OpenFlow tables to