It happens on my H120 too. This with a low-bitrate file, needing 10-15% boost to play (using default settings). It looks like the codec thread is blocked from reading while the disk spins up, causing the PCM buffer to run dry. That enabling crossfade fixes it makes sense then, since the PCM buffer is much larger in that case.
It wouldn't surprise me if this affects most/all swcodec targets with a hard drive...