CDs are 44.1 kHz. If you rip a flac from a CD, and play it through the optical out to a DAC, if the rip is perfect, you will get the *exact* same audio that you get from a CD player with optical out.
Because it is digital output, the fault lies with the file, not the hardware, unless either there's a bug in Rockbox right now, or you had the equalizer, crossfeed, or some other sound-coloring feature enabled.