ovn-controller: Handle physical changes correctly

ovn-controller: Handle physical changes correctly

[1] reported increased failure rates in certain tests
with incremental processing (the numbers are the number of failures
seen in 100 tests):

   2  ovn -- vtep: 3 HVs, 1 VIFs/HV, 1 GW, 1 LS
  10  ovn -- 2 HVs, 2 LS, 1 lport/LS, 2 peer LRs
  52  ovn -- 1 HV, 1 LS, 2 lport/LS, 1 LR
  45  ovn -- 1 HV, 2 LSs, 1 lport/LS, 1 LR
  23  ovn -- 2 HVs, 3 LS, 1 lport/LS, 2 peer LRs, static routes
  53  ovn -- 2 HVs, 3 LRs connected via LS, static routes
  32  ovn -- 2 HVs, 2 LRs connected via LS, gateway router
  50  ovn -- icmp_reply: 1 HVs, 2 LSs, 1 lport/LS, 1 LR

These failures were caused by a combination of problems in
handling physical changes:

  1. When a vif was removed, the localvif_to_ofport entry was not
     removed.
  2. When a physical change was detected, ovn-controller would wait
     a poll cycle before processing the logical flow table.

This patch set addresses both of these issues while simultaneously
cleaning up the code in physical.c.  A side effect is a modification
of where OF flows are dumped in the gateway router case that allowed
the root causes of this issue to be found.

With these changes, all of the above tests had a 100/100 success rate.

[1] http://openvswitch.org/pipermail/dev/2016-July/075803.html

Signed-off-by: Ryan Moats <rmoats@us.ibm.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>