KVM: Don't keep reference to irq routing table in irqfd struct