IPoIB: Avoid flushing the driver workqueue on dev_down