PCI: rcar: Don't disable/unprepare clocks on prepare/enable failure