ovn-controller: Handle physical changes correctly