Rockbox Audio Quality versus Original Firmware (RMAA comparison)

[Febs]

I used Rightmark Audio Analyzer to compare Rockbox versus the original software.

The results are set forth in successive posts below.

All tests were performed on my 2.8gHz Pentium 4 PC with an Audigy 2 sound card.  Other test conditions are noted in the individual posts below.

[Febs]

Rockbox versus original Apple firmware.
iPod video 60GB
Headphone output
Volume at 0dB in Rockbox, matched as closely as possible in the Apple FW using RMAA's level check.

Rockbox frequency response (headphone output):



Rockbox frequency response (line output using Pocket Dock):




Apple firmware frequency response:



Rockbox dynamic range:



Apple FW dynamic range:

[Febs]

I have seen speculation that the decoders in the respective firmwares could cause differences in sound, so I encoded the RMAA test sound file from .wav to .mp3 using LAME 3.97b2 -V2.  All other test conditions are the same as above.

Rockbox frequency response (MP3, headphone output):




Rockbox frequency response (MP3, line output):



Apple FW frequency response (MP3,headphone output):



The RMAA test uses a sine wave sweep, which is notoriously difficult for mp3 encoders to handle, so I'm not sure how valuable this test is.  Nevertheless, it is interesting to see that the results are substantially similar for both Rockbox and the original Apple firmware.

[lachlan]

I would go as far as to say they are almost identical. Out of curiosity, would it better to have the line showing greater changes, or being smoother?

[omendelovitz]

Hey Febs, a quick Q 4 U.  What other measures of sound output would one consider in assessing SQ?  I ask b/c my main beef w/ the original OS is that it seemed to sound more "congested" than Rockbox, i.e. worse instrument separation, resolution (layering) and imaging.  I'm also wondering about the background hiss audible in AOS w/ high sensitivity IEM's.  

I'm not trying to be facetious here, just want to learn...

[aegis]

Hi Febs,

I'm no engineer but these graphs look pretty the same and the tiny differences can be just statistically irrelevant so I would ask: how much are frequency response and dynamics just a question of hardware - quality of filters, circuits etc?

If the mp3 file is the same the only difference between systems is the decoder used by particular players and equallizer presets. Can someone explain if it has anything to do with frequency response assuming codec is not just broken?

[Febs]

I think that these are all good questions, but I am no engineer either.  I would love to get input from some of the more technical people here as to how these results should be interpreted.

omendelovitz, are you certain that you are comparing the two with the exact same volume and with no processing or EQ involved?

I've added a frequency response graph of Rockbox/iPod with the line-out to the second post above.

[omendelovitz]

Yeah, Febs I am.  I should Qualify last night's statement though.  Truthfully, using a VERY transparent Amp and High End Cans, I played my reference tunes through both RB and AOS, one after another (admittedly, with a pause = reboot time).  My new conclusions are:  RB makes iPod sound more like ZVM - wider soundstage (and almost by extension, better imaging - to much to explain and I'd probably mess it up, so I won't go there); otherwise, the two sound the same.  A proper analog for this scenario would be Grado sound vs. Sennheiser sound.  

Hope that helps.

[Febs]

Unfortunately, I could see this thread getting bogged down with debate regarding wholly subjective impressions of sound quality, and since there are other existing threads on that, I think that we should not go there.  The whole purpose of this thread was to take an objective look at the differences between the firmware.  So, for purposes of this thread, let's stick with objective analysis.

(I will say this:  if you're going to analogize between the Sennheiser and Grado sound, I think that even the staunchest subjectivist would agree that the Sennheiser "veil" should be observable in the frequency response.)

[breez]

Have you tried the stereo crosstalk test in RMAA and found any differences between firmwares?

[omendelovitz]

I agree, but I don't think Sennheiser has a veil.  In that case, let's suffice to say that RB sounds more like a concert hall or stadium in terms of soundstage width, and AOS sounds more like being on stage at a rock concert or small club.  Like I said earlier, the difference in sound is comparable to ABX'ing a creative Zen Vision M (line out) to the AOS (line out).  i.e. RB presents the soundstage the same as Creative does in the Zen Vision M.

'nuff said

[Febs]

Like I said earlier, the difference in sound is comparable to ABX'ing a creative Zen Vision M (line out) to the AOS (line out).

omen, I would agree with you if your comparison had involved an ABX test.  I think that it would be *very* interesting to do an ABX test.

Have you tried the stereo crosstalk test in RMAA and found any differences between firmwares?

I had RMMA run all of the available tests.  Give me a minute and I will post the saved RMAA files here.

Edit:  Here they are:

iriver headphone out, Rockbox
ipod, headphone out, Apple fw, mp3
iPod headphone out, Apple fw
iPod headphone out, Rockbox, mp3
iPod headphone out, Rockbox
iPod line out, Rockbox

You should be able to open these files in RMAA and view all of the results.

[saratoga]

Yeah, Febs I am.  I should Qualify last night's statement though.  Truthfully, using a VERY transparent Amp and High End Cans, I played my reference tunes through both RB and AOS, one after another (admittedly, with a pause = reboot time).  My new conclusions are:  RB makes iPod sound more like ZVM - wider soundstage (and almost by extension, better imaging - to much to explain and I'd probably mess it up, so I won't go there); otherwise, the two sound the same.  A proper analog for this scenario would be Grado sound vs. Sennheiser sound.  

Hope that helps.

Try an ABX test.  The stereo seperation between the two is virtually identical, so you shouldn't be hearing anything at all.  My guess is the delay between listening to the two samples and the placebo is effecting you.

I'm no engineer but these graphs look pretty the same and the tiny differences can be just statistically irrelevant so I would ask: how much are frequency response and dynamics just a question of hardware - quality of filters, circuits etc?

Its basically all hardware, at least for lossless tests.  You'll only see a difference if something is very wrong (like we did on the X5 a while back when it was accidently put in 22.05k sample rate instead of 44.1k).  Thats why the Rockbox and Apple firmwares perform the same for the WAV test.

The lossy tests are effected by the decoder.

What other measures of sound output would one consider in assessing SQ?

Noise floor will tell you how bad the hiss on the system will be.  Frequency response will tell you how much EQ you'll need.  THD and IMD will tell you how much distortion there in the system.  Crosstalk tells you how well stereo works.  No one of them alone is more important then the rest, it just depends on what you need to know.

Also, be careful with the frequency response plot.  That is valid ONLY for something with the same load impedance as the test setup.  For a sound card, the impedance will be VERY high, while typical headphones are very low.  So if you want to test frequency response as you'll hear it, you should test with your specific headphones in parallel with the sound card (doing this on the ipod gives surprisingly different results with <32 ohm headphones too).

[omendelovitz]

If anyone is in the Toronto, Canada area and has a 5G ipod they want to loan me and/or come test with me, I'd be glad to ABX.  As it stands right now, I don't have that kind of equipment.  I'd love to try this option as well.

[saratoga]

If anyone is in the Toronto, Canada area and has a 5G ipod they want to loan me and/or come test with me, I'd be glad to ABX.  As it stands right now, I don't have that kind of equipment.  I'd love to try this option as well.

If you don't have an Ipod, how did you manage to do the previous test?

Edit:  Do you mean you want two ipods to do an ABX test?  Thats not needed, or even a good idea.  Just record the output of the firmwares using your sound card.  Then load them up in any wave editor and make sure to clip the wavs so they start and end on exactly the same point in the song.  FInally, load into any ABX program (foobar w/ ABX plugin, winabx, HR/ABC, etc).  Since the output appears to be identical, you should get essentially identical recordings.