Documentation: DPDK IVSHMEM VM Communications

Adds documentation on how to run IVSHMEM communication through VM.

Signed-off-by: Mike A. Polehn <mike.a.polehn@intel.com>
Acked-by: Pravin B Shelar <pshelar@nicira.com>

diff --git a/INSTALL.DPDK b/INSTALL.DPDK
index ff729e3..bba7d29 100644 (file)
--- a/INSTALL.DPDK
+++ b/INSTALL.DPDK
@@ -20,9 +20,13 @@ DPDK:
 Set dir i.g.:   export DPDK_DIR=/usr/src/dpdk-1.7.0
 cd $DPDK_DIR
 update config/common_linuxapp so that dpdk generate single lib file.
 Set dir i.g.:   export DPDK_DIR=/usr/src/dpdk-1.7.0
 cd $DPDK_DIR
 update config/common_linuxapp so that dpdk generate single lib file.

+(modification also required for IVSHMEM build)

 CONFIG_RTE_BUILD_COMBINE_LIBS=y
 
 CONFIG_RTE_BUILD_COMBINE_LIBS=y
 

+For default install without IVSHMEM:

 make install T=x86_64-native-linuxapp-gcc
 make install T=x86_64-native-linuxapp-gcc

+To include IVSHMEM (shared memory):
+make install T=x86_64-ivshmem-linuxapp-gcc

 For details refer to  http://dpdk.org/
 
 Linux kernel:
 For details refer to  http://dpdk.org/
 
 Linux kernel:


@@ -30,9 +34,13 @@ Refer to intel-dpdk-getting-started-guide.pdf for understanding
 DPDK kernel requirement.
 
 OVS:
 DPDK kernel requirement.
 
 OVS:

+Non IVSHMEM:
+export DPDK_BUILD=$DPDK_DIR/x86_64-native-linuxapp-gcc/
+IVSHMEM:
+export DPDK_BUILD=$DPDK_DIR/x86_64-ivshmem-linuxapp-gcc/
+

 cd $(OVS_DIR)/openvswitch
 ./boot.sh
 cd $(OVS_DIR)/openvswitch
 ./boot.sh

-export DPDK_BUILD=$DPDK_DIR/x86_64-native-linuxapp-gcc/

 ./configure --with-dpdk=$DPDK_BUILD
 make
 
 ./configure --with-dpdk=$DPDK_BUILD
 make
 


@@ -73,7 +81,7 @@ First setup DPDK devices:
 
 Prepare system:
   - mount hugetlbfs
 
 Prepare system:
   - mount hugetlbfs

-    e.g. mount -t hugetlbfs -o pagesize=1G none /mnt/huge/
+    e.g. mount -t hugetlbfs -o pagesize=1G none /dev/hugepages

 
 Ref to http://www.dpdk.org/doc/quick-start for verifying DPDK setup.
 
 
 Ref to http://www.dpdk.org/doc/quick-start for verifying DPDK setup.
 


@@ -91,7 +99,7 @@ Start ovsdb-server as discussed in INSTALL doc:
       ./ovsdb/ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \
           --remote=db:Open_vSwitch,Open_vSwitch,manager_options \
           --private-key=db:Open_vSwitch,SSL,private_key \
       ./ovsdb/ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \
           --remote=db:Open_vSwitch,Open_vSwitch,manager_options \
           --private-key=db:Open_vSwitch,SSL,private_key \

-          --certificate=dbitch,SSL,certificate \
+          --certificate=Open_vSwitch,SSL,certificate \

           --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach
     First time after db creation, initialize:
       cd $OVS_DIR
           --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach
     First time after db creation, initialize:
       cd $OVS_DIR


@@ -105,12 +113,13 @@ for dpdk initialization.
 
    e.g.
    export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
 
    e.g.
    export DB_SOCK=/usr/local/var/run/openvswitch/db.sock

-   ./vswitchd/ovs-vswitchd --dpdk -c 0x1 -n 4 -- unix:$DB_SOCK  --pidfile --detach
+   ./vswitchd/ovs-vswitchd --dpdk -c 0x1 -n 4 -- unix:$DB_SOCK --pidfile --detach

 
 

-If allocated more than 1 GB huge pages, set amount and use NUMA node 0 memory:
+If allocated more than one GB hugepage (as for IVSHMEM), set amount and use NUMA
+node 0 memory:

 
    ./vswitchd/ovs-vswitchd --dpdk -c 0x1 -n 4 --socket-mem 1024,0 \
 
    ./vswitchd/ovs-vswitchd --dpdk -c 0x1 -n 4 --socket-mem 1024,0 \

-      -- unix:$DB_SOCK  --pidfile --detach
+      -- unix:$DB_SOCK --pidfile --detach

 
 To use ovs-vswitchd with DPDK, create a bridge with datapath_type
 "netdev" in the configuration database.  For example:
 
 To use ovs-vswitchd with DPDK, create a bridge with datapath_type
 "netdev" in the configuration database.  For example:


@@ -136,9 +145,7 @@ Test flow script across NICs (assuming ovs in /usr/src/ovs):
 ############################# Script:
 
 #! /bin/sh
 ############################# Script:
 
 #! /bin/sh

-

 # Move to command directory
 # Move to command directory

-

 cd /usr/src/ovs/utilities/
 
 # Clear current flows
 cd /usr/src/ovs/utilities/
 
 # Clear current flows


@@ -158,7 +165,8 @@ help.
 
 At this time all ovs-vswitchd tasks end up being affinitized to cpu core 0
 but this may change. Lets pick a target core for 100% task to run on, i.e. core 7.
 
 At this time all ovs-vswitchd tasks end up being affinitized to cpu core 0
 but this may change. Lets pick a target core for 100% task to run on, i.e. core 7.

-Also assume a dual 8 core sandy bridge system with hyperthreading enabled.
+Also assume a dual 8 core sandy bridge system with hyperthreading enabled
+where CPU1 has cores 0,...,7 and 16,...,23 & CPU2 cores 8,...,15 & 24,...,31.

 (A different cpu configuration will have different core mask requirements).
 
 To give better ownership of 100%, isolation maybe useful.
 (A different cpu configuration will have different core mask requirements).
 
 To give better ownership of 100%, isolation maybe useful.


@@ -178,11 +186,11 @@ taskset -p 080 1762
   pid 1762's new affinity mask: 80
 
 Assume that all other ovs-vswitchd threads to be on other socket 0 cores.
   pid 1762's new affinity mask: 80
 
 Assume that all other ovs-vswitchd threads to be on other socket 0 cores.

-Affinitize the rest of the ovs-vswitchd thread ids to 0x0FF007F
+Affinitize the rest of the ovs-vswitchd thread ids to 0x07F007F

 
 

-taskset -p 0x0FF007F {thread pid, e.g 1738}
+taskset -p 0x07F007F {thread pid, e.g 1738}

   pid 1738's current affinity mask: 1
   pid 1738's current affinity mask: 1

-  pid 1738's new affinity mask: ff007f
+  pid 1738's new affinity mask: 7f007f

 . . .
 
 The core 23 is left idle, which allows core 7 to run at full rate.
 . . .
 
 The core 23 is left idle, which allows core 7 to run at full rate.


@@ -207,8 +215,8 @@ with the ring naming used within ovs.
 location tests/ovs_client
 
 To run the client :
 location tests/ovs_client
 
 To run the client :

-
-    ovsclient -c 1 -n 4 --proc-type=secondary -- -n "port id you gave dpdkr"
+  cd /usr/src/ovs/tests/
+  ovsclient -c 1 -n 4 --proc-type=secondary -- -n "port id you gave dpdkr"

 
 In the case of the dpdkr example above the "port id you gave dpdkr" is 0.
 
 
 In the case of the dpdkr example above the "port id you gave dpdkr" is 0.
 


@@ -218,6 +226,9 @@ The application simply receives an mbuf on the receive queue of the
 ethernet ring and then places that same mbuf on the transmit ring of
 the ethernet ring.  It is a trivial loopback application.
 
 ethernet ring and then places that same mbuf on the transmit ring of
 the ethernet ring.  It is a trivial loopback application.
 

+DPDK rings in VM (IVSHMEM shared memory communications)
+-------------------------------------------------------
+

 In addition to executing the client in the host, you can execute it within
 a guest VM. To do so you will need a patched qemu.  You can download the
 patch and getting started guide at :
 In addition to executing the client in the host, you can execute it within
 a guest VM. To do so you will need a patched qemu.  You can download the
 patch and getting started guide at :