arm64: unmap idmap earlier