drm: Refactor drop/set master code a bit