clk: at91: Revert "keep slow clk enabled to prevent system hang"