net: Remove the now unused vrf_ptr