ASoC: core: Clean up DAPM before the card debugfs