atmel_serial: keep clock off when it's not needed