net/mlx4_core: Adjust port number in qp_attach wrapper when detaching