RDMA/nes: Fix crash when listener destroyed during loopback setup