IB/IPoIB: Fix race between ipoib_remove_one to sysfs functions