1 OVS-on-Hyper-V Design Document
2 ==============================
3 There has been an effort in the recent past to develop the Open vSwitch (OVS)
4 solution onto multiple hypervisor platforms such as FreeBSD and Microsoft
5 Hyper-V. VMware has been working on a OVS solution for Microsoft Hyper-V for
6 the past few months and has successfully completed the implementation.
8 This document provides details of the development effort. We believe this
9 document should give enough information to members of the community who are
10 curious about the developments of OVS on Hyper-V. The community should also be
11 able to get enough information to make plans to leverage the deliverables of
14 The userspace portion of the OVS has already been ported to Hyper-V and
15 committed to the openvswitch repo. So, this document will mostly emphasize on
16 the kernel driver, though we touch upon some of the aspects of userspace as
19 We cover the following topics:
20 1. Background into relevant Hyper-V architecture
21 2. Design of the OVS Windows implementation
22 a. Kernel module (datapath)
23 b. Userspace components
24 c. Kernel-Userspace interface
26 3. Build/Deployment environment
28 For more questions, please contact dev@openvswitch.org
30 1) Background into relevant Hyper-V architecture
31 ------------------------------------------------
32 Microsoft’s hypervisor solution - Hyper-V[1] implements a virtual switch that
33 is extensible and provides opportunities for other vendors to implement
34 functional extensions[2]. The extensions need to be implemented as NDIS drivers
35 that bind within the extensible switch driver stack provided. The extensions
36 can broadly provide the functionality of monitoring, modifying and forwarding
37 packets to destination ports on the Hyper-V extensible switch. Correspondingly,
38 the extensions can be categorized into the following types and provide the
40 * Capturing extensions: monitoring packets
41 * Filtering extensions: monitoring, modifying packets
42 * Forwarding extensions: monitoring, modifying, forwarding packets
44 As can be expected, the kernel portion (datapath) of OVS on Hyper-V solution
45 will be implemented as a forwarding extension.
47 In Hyper-V, the virtual machine is called the Child Partition. Each VIF or
48 physical NIC on the Hyper-V extensible switch is attached via a port. Each port
49 is both on the ingress path or the egress path of the switch. The ingress path
50 is used for packets being sent out of a port, and egress is used for packet
51 being received on a port. By design, NDIS provides a layered interface, where
52 in the ingress path, higher level layers call into lower level layers, and on
53 the egress path, it is the other way round. In addition, there is a object
54 identifier (OID) interface for control operations Eg. addition of a port. The
55 workflow for the calls is similar in nature to the packets, where higher level
56 layers call into the lower level layers. A good representational diagram of
57 this architecture is in [4].
59 Windows Filtering Platform (WFP)[5] is a platform implemented on Hyper-V that
60 provides APIs and services for filtering packets. WFP has been utilized to
61 filter on some of the packets that OVS is not equipped to handle directly. More
62 details in later sections.
64 IP Helper [6] is a set of API available on Hyper-V to retrieve information
65 related to the network configuration information on the host machine. IP Helper
66 has been used to retrieve some of the configuration information that OVS needs.
69 2) Design of the OVS Windows implementation
70 -------------------------------------------
72 +-------------------------------+
76 +------+ +--------------+ | +-----------+ +------------+ |
78 | OVS- | | OVS | | | Virtual | | Virtual | |
79 | wind | | USERSPACE | | | Machine #1| | Machine #2 | |
80 | | | DAEMON/CTL | | | | | | |
81 +------+-++---+---------+ | +--+------+-+ +----+------++ | +--------+
82 | DPIF- | | netdev- | | |VIF #1| |VIF #2| | |Physical|
83 | Windows |<=>| Windows | | +------+ +------+ | | NIC |
84 +---------+ +---------+ | || /\ | +--------+
85 User /\ | || *#1* *#4* || | /\
86 =========||=======================+------||-------------------||--+ ||
87 Kernel || \/ || ||=====/
88 \/ +-----+ +-----+ *#5*
89 +-------------------------------+ | | | |
90 | +----------------------+ | | | | |
91 | | OVS Pseudo Device | | | | | |
92 | +----------------+-----+ | | | | |
94 | +------------+ | | N | | E |
95 | | Flowtable | +------------+ | | G | | G |
96 | +------------+ | Packet | |*#2*| R | | R |
97 | +--------+ | Processing | |<=> | E | | E |
98 | | WFP | | | | | S | | S |
99 | | Driver | +------------+ | | S | | S |
100 | +--------+ | | | | |
102 | OVS FORWARDING EXTENSION | | | | |
103 +-------------------------------+ +-----+-----------------+-----+
104 |HYPER-V Extensible Switch *#3|
105 +-----------------------------+
108 Fig 2. Various blocks of the OVS Windows implementation
110 Figure 2 shows the various blocks involved in the OVS Windows implementation,
111 along with some of the components available in the NDIS stack, and also the
112 virtual machines. The workflow of a packet being transmitted from a VIF out and
113 into another VIF and to a physical NIC is also shown. New userspace components
114 being added as also shown. Later on in this section, we’ll discuss the flow of
115 a packet at a high level.
117 The figure gives a general idea of where the OVS userspace and the kernel
118 components fit in, and how they interface with each other.
120 The kernel portion (datapath) of OVS on Hyper-V solution has be implemented as
121 a forwarding extension roughly implementing the following
122 sub-modules/functionality. Details of each of these sub-components in the
123 kernel are contained in later sections:
124 * Interfacing with the NDIS stack
125 * Switch/Datapath management
126 * Interfacing with userspace portion of the OVS solution to implement the
127 necessary ioctls that userspace needs
129 * Flowtable/Actions/packet forwarding
131 * Event notifications
133 The datapath for the OVS on Linux is a kernel module, and cannot be directly
134 ported since there are significant differences in architecture even though the
135 end functionality provided would be similar. Some examples of the differences
137 * Interfacing with the NDIS stack to hook into the NDIS callbacks for
138 functionality such as receiving and sending packets, packet completions,
139 OIDs used for events such as a new port appearing on the virtual switch.
140 * Interface between the userspace and the kernel module.
141 * Event notifications are significantly different.
142 * The communication interface between DPIF and the kernel module need not be
143 implemented in the way OVS on Linux does.
144 * Any licensing issues of using Linux kernel code directly.
146 Due to these differences, it was a straightforward decision to develop the
147 datapath for OVS on Hyper-V from scratch rather than porting the one on Linux.
148 A re-development focussed on the following goals:
149 * Adhere to the existing requirements of userspace portion of OVS (such as
150 ovs- vswitchd), to minimize changes in the userspace workflow.
151 * Fit well into the typical workflow of a Hyper-V extensible switch forwarding
154 The userspace portion of the OVS solution is mostly POSIX code, and not very
155 Linux specific. Majority of the code has already been ported and committed to
156 the openvswitch repo. Most of the daemons such as ovs-vswitchd or ovsdb-server
157 can run on Windows now. One additional daemon that has been implemented is
158 called ovs-wind. At a high level ovs-wind manages keeps the ovsdb used by
159 userspace in sync with the kernel state. More details in the userspace section.
161 As explained in the OVS porting design document [7], DPIF is the portion of
162 userspace that interfaces with the kernel portion of the OVS. Each platform can
163 have its own implementation of the DPIF provider whose interface is defined in
164 dpif-provider.h [3]. For OVS on Hyper-V, we have an implementation of DPIF
165 provider for Hyper-V. The communication interface between userspace and the
166 kernel is a pseudo device and is different from that of the Linux’s DPIF
167 provider which uses netlink. But, as long as the DPIF provider interface is the
168 same, the callers should be agnostic of the underlying communication interface.
170 2.a) Kernel module (datapath)
171 -----------------------------
173 Interfacing with the NDIS stack
174 -------------------------------
175 For each virtual switch on Hyper-V, the OVS extensible switch extension can be
176 enabled/disabled. We support enabling the OVS extension on only one switch.
177 This is consistent with using a single datapath in the kernel on Linux. All the
178 physical adapters are connected as external adapters to the extensible switch.
180 When the OVS switch extension registers itself as a filter driver, it also
181 registers callbacks for the switch management and datapath functions. In other
182 words, when a switch is created on the Hyper-V root partition (host), the
183 extension gets an activate callback upon which it can initialize the data
184 structures necessary for OVS to function. Similarly, there are callbacks for
185 when a port gets added to the Hyper-V switch, and an External Network adapter
186 or a VM Network adapter is connected/disconnected to the port. There are also
187 callbacks for when a VIF (NIC of a child partition) send out a packet, or a
188 packet is received on an external NIC.
190 As shown in the figures, an extensible switch extension gets to see a packet
191 sent by the VM (VIF) twice - once on the ingress path and once on the egress
192 path. Forwarding decisions are to be made on the ingress path. Correspondingly,
193 we’ll be hooking onto the following interfaces:
194 * Ingress send indication: intercept packets for performing flow based
195 forwarding.This includes straight forwarding to output ports. Any packet
196 modifications needed to be performed are done here either inline or by
197 creating a new packet. A forwarding action is performed as the flow actions
199 * Ingress completion indication: cleanup and free packets that we generated on
200 the ingress send path, pass-through for packets that we did not generate.
201 * Egress receive indication: pass-through.
202 * Egress completion indication: pass-through.
204 Interfacing with OVS userspace
205 ------------------------------
206 We’ve implemented a pseudo device interface for letting OVS userspace talk to
207 the OVS kernel module. This is equivalent to the typical character device
208 interface on POSIX platforms. The pseudo device supports a whole bunch of
209 ioctls that netdev and DPIF on OVS userspace make use of.
211 Switch/Datapath management
212 --------------------------
213 As explained above, we hook onto the management callback functions in the NDIS
214 interface for when to initialize the OVS data structures, flow tables etc. Some
215 of this code is also driven by OVS userspace code which sends down ioctls for
216 operations like creating a tunnel port etc.
220 As explained above, we hook onto the management callback functions in the NDIS
221 interface to know when a port is added/connected to the Hyper-V switch. We use
222 these callbacks to initialize the port related data structures in OVS. Also,
223 some of the ports are tunnel ports that don’t exist on the Hyper-V switch that
224 are initiated from OVS userspace.
226 Flowtable/Actions/packet forwarding
227 -----------------------------------
228 The flowtable and flow actions based packet forwarding is the core of the OVS
229 datapath functionality. For each packet on the ingress path, we consult the
230 flowtable and execute the corresponding actions. The actions can be limited to
231 simple forwarding to a particular destination port(s), or more commonly
232 involves modifying the packet to insert a tunnel context or a VLAN ID, and
233 thereafter forwarding to the external port to send the packet to a destination
238 We make use of the Internal Port on a Hyper-V switch for implementing
239 tunneling. The Internal Port is a virtual adapter that is exposed on the Hyper-
240 V host, and connected to the Hyper-V switch. Basically, it is an interface
241 between the host and the virtual switch. The Internal Port acts as the Tunnel
242 end point for the host (aka VTEP), and holds the VTEP IP address.
244 Tunneling ports are not actual ports on the Hyper-V switch. These are virtual
245 ports that OVS maintains and while executing actions, if the outport is a
246 tunnel port, we short circuit by performing the encapsulation action based on
247 the tunnel context. The encapsulated packet gets forwarded to the external
248 port, and appears to the outside world as though it was set from the VTEP.
250 Similarly, when a tunneled packet enters the OVS from the external port bound
251 to the internal port (VTEP), and if yes, we short circuit the path, and
252 directly forward the inner packet to the destination port (mostly a VIF, but
253 dictated by the flow). We leverage the Windows Filtering Platform (WFP)
254 framework to be able to receive tunneled packets that cannot be decapsulated by
255 OVS right away. Currently, fragmented IP packets fall into that category, and
256 we leverage the code in the host IP stack to reassemble the packet, and
257 performing decapsulation on the reassembled packet.
259 We’ll also be using the IP helper library to provide us IP address and other
260 information corresponding to the Internal port.
264 The pseudo device interface described above is also used for providing event
265 notifications back to OVS userspace. A shared memory/overlapped IO model is
268 2.b) Userspace components
269 -------------------------
270 A new daemon has been added to userspace to manage the entities in OVSDB, and
271 also to keep it in sync with the kernel state, and this include bridges,
272 physical NICs, VIFs etc. For example, upon bootup, ovs-wind does a get on the
273 kernel to get a list of the bridges, and the corresponding ports and populates
274 OVSDB. If a new VIF gets added to the kernel switch because a user powered on a
275 Virtual Machine, ovs-wind detects it, and adds a corresponding entry in the
276 ovsdb. This implies that ovs-wind has a synchronous as well as an asynchronous
277 interface to the OVS kernel driver.
280 2.c) Kernel-Userspace interface
281 -------------------------------
284 DPIF-Windows is the Windows implementation of the interface defined in dpif-
285 provider.h, and provides an interface into the OVS kernel driver. We implement
286 most of the callbacks required by the DPIF provider. A quick summary of the
287 functionality implemented is as follows:
288 * dp_dump, dp_get: dump all datapath information or get information for a
289 particular datapath. Currently we only support one datapath.
290 * flow_dump, flow_put, flow_get, flow_flush: These functions retrieve all
291 flows in the kernel, add a flow to the kernel, get a specific flow and
292 delete all the flows in the kernel.
293 * recv_set, recv, recv_wait, recv_purge: these poll packets for upcalls.
294 * execute: This is used to send packets from userspace to the kernel. The
295 packets could be either flow miss packet punted from kernel earlier or
296 userspace generated packets.
297 * vport_dump, vport_get, ext_info: These functions dump all ports in the
298 kernel, get a specific port in the kernel, or get extended information
300 * event_subscribe, wait, poll: These functions subscribe, wait and poll the
301 events that kernel posts. A typical example is kernel notices a port has
302 gone up/down, and would like to notify the userspace.
306 We have a Windows implementation of the the interface defined in lib/netdev-
307 provider.h. The implementation provided functionality to get extended
308 information about an interface. It is limited in functionality compared to the
309 Linux implementation of the netdev provider and cannot be used to add any
310 interfaces in the kernel such as a tap interface.
313 2.d) Flow of a packet
314 ---------------------
315 Figure 2 shows the numbered steps in which a packets gets sent out of a VIF and
316 is forwarded to another VIF or a physical NIC. As mentioned earlier, each VIF
317 is attached to the switch via a port, and each port is both on the ingress and
318 egress path of the switch, and depending on whether a packet is being
319 transmitted or received, one of the paths gets used. In the figure, each step n
322 The steps are as follows:
323 1. When a packet is sent out of a VIF or an physical NIC or an internal port,
324 the packet is part of the ingress path.
325 2. The OVS kernel driver gets to intercept this packet.
326 a. OVS looks up the flows in the flowtable for this packet, and executes the
327 corresponding action.
328 b. If there is not action, the packet is sent up to OVS userspace to examine
329 the packet and figure out the actions.
330 v. Userspace executes the packet by specifying the actions, and might also
331 insert a flow for such a packet in the future.
332 d. The destination ports are added to the packet and sent down to the Hyper-
334 3. The Hyper-V forwards the packet to the destination ports specified in the
335 packet, and sends it out on the egress path.
336 4. The packet gets forwarded to the destination VIF.
337 5. It might also get forwarded to a physical NIC as well, if the physical NIC
338 has been added as a destination port by OVS.
343 The userspace components added as part of OVS Windows implementation have been
344 integrated with autoconf, and can be built using the steps mentioned in the
345 BUILD.Windows file. Additional targets need to be specified to make.
347 The OVS kernel code is part of a Visual Studio 2013 solution, and is compiled
348 from the IDE. There are plans in the future to move this to a compilation mode
349 such that we can compile it without an IDE as well.
351 Once compiled, we have an install script that can be used to load the kernel
357 1: Hyper-V Extensible Switch
358 http://msdn.microsoft.com/en-us/library/windows/hardware/hh598161(v=vs.85).aspx
359 2: Hyper-V Extensible Switch Extensions
360 http://msdn.microsoft.com/en-us/library/windows/hardware/hh598169(v=vs.85).aspx
362 http://openvswitch.sourcearchive.com/documentation/1.1.0-1/dpif-
363 provider_8h_source.html
364 4. Hyper-V Extensible Switch Components
365 http://msdn.microsoft.com/en-us/library/windows/hardware/hh598163(v=vs.85).aspx
366 5. Windows Filtering Platform
367 http://msdn.microsoft.com/en-us/library/windows/desktop/aa366510(v=vs.85).aspx
369 http://msdn.microsoft.com/en-us/library/windows/hardware/ff557015(v=vs.85).aspx
370 7. How to Port Open vSwitch to New Software or Hardware
371 http://git.openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob;f=PORTING
projects / cascardo / ovs.git / blob