netfilter: nft_hash: fix hash overflow validation