cxl: Fix issues when unmapping contexts