Rockbox Audio Quality versus Original Firmware (RMAA comparison)

[travishayes89]

i have not run such tests, but i have listened to music through rb on my ipod (before it got dunked) and also on the apple os, and noticed that the rb output sounded better on certain songs

[ryran]

No way in hell I could abx those two (I gave it a try).

[In other news: in la futura, why not just use wavgain to get the samples to exactly the same volume?]

[omendelovitz]

Febs, Saratoga,

EDIT:  After checking all my sound settings, I realized that while I thought I had all sound effects off, it turns out that in fact I didn't.  Upon a preliminary re-listen, I am finding a diminishing difference b/w the two OS'.  OOPS!  How embarrassing...  Anyway, I will listen to your ABX samples, as soon as I learn how to ABX  

Saratoga, thanks for the details; Febs, thanks for the tests - anywhere you know where I can find instructions on Foobar ABX'ing?  I'll check the help file first I guess.

[soap]

(Assuming Foobar 0.8.3) (Assuming you have the ABX component installed) (Believe it comes with the Special and Full builds)
1- Drop the two files into Foobar.
2- Select the two tracks, right click, and from the context menu select "ABX two tracks"
3- Foobar will do its thing preparing the files, then will present you with the ABX comparator screen.  From here you can listen to the "A" sample, the "B" sample, the "X" sample, and the "Y" sample.  The letters assigned to your two tracks will be randomly changed each trial, your task is to play the files however you want and tell if A=X or if A=Y.
The prefered methodology is to check the "Hide results" selection box, and to do at least 12 "trials" before exiting and saving your results as a text file.

[XavierGr]

Bravo Febs! Magnificent work on the samples and analyzing both firmware output!
This is a definite benchmark to shut off the mouths of some members here and MisticRiver.

As many of you have noticed, Rockbox is being under "attack" as having "inferior" sound quality compared to original firmware. (see the first golden quote in wiki: http://www.rockbox.org/twiki/bin/view/Main/GoldenQuotes)

For once again these persons are proven wrong. These tests show that Rockbox differences from original firmware are inaudible.

Again, thank you Febs, for providing the most failsafe way to compare sound quality and for the analysis that you made.
Let’s hope that this debate will end, and Rockbox will no longer be under the "inferior-sound-quality" myth.

PS: I will wait for omendelovitz's ABX test before continuing my "triumphology", who knows....  

[omendelovitz]

Science overall.  I have embraced my inner scientist and resigned from audiophiledom, possibly permanently.  

Just a few quips here before I move on to my results:

1 - Foobar's ABX is an absolute wonder!  I love this program.  It is the cream of software, forget the fact that it is freeware...

2 - There is truly nothing better than being able to test one's own biases and perceptions in this manner;

3 - I can't wait for Rockbox to come out in full release for the iPod.  As a FW overall, it really trumps Apple.

Now for my results [drum roll please] - with my soundcard, high end headphones and crystal clear and very transparent amp I...

can tell no difference b/w the two wav files febs so graciously posted online for me.  Now, before you start triumphonolizing, Xavier, I must add a "scientist's" warning to my findings - I would personally claim that therefore there is no audible difference b/w the two firmwares, using Febs' readings in conjunction with my observations, but in reality, my soundcard is internal and therefore isn't all that great.  I will also add that when comparing lossless to mp3 alt insane, I could hear no difference on the same setup.  Take that for what you will.  

What I will say [finally] are two last things - 1) kudos to Febs for his work and perseverance; 2) if anyone wants to make any claims re: RB SQ to AFW, just point them to this thread and tell them to put out or shut up! And then ignore those people, b/c they're going to see what they want to see (and disregard the rest... thx for the quote, Paul Simon!).

Over and Out - Omendelovitz

[Atheistic Freedom]

Yeah, but turn on Hardware EQ and Software EQ Precut to -12 dB...  

Worlds better.

[XavierGr]

Omendelovitz, thanks for taking the test. You are the first one to take it, that is admirable.

Of course I didn't mean any direct offence at you. You said that finally, it was a weird EQ setup that made you believe Rockbox sounded different.

Anyway, I am a happy man now!

P.S: I am so happy that I will sticky this.

[Llorean]

Stickying should be reserved for forum business. If sound quality comparison needs to be put in a permanent place, it should be made into a wiki page.

A link to that wiki page could be incorporated into a playback FAQ, but this isn't really something that belongs stickied on its own, as it'd be best if there were a minimum of stickies.

[wolfB]

I'd agree the soundquality of the dock output from a 5G iPod can be considered equal. But I can definitely recognize a difference from the headphone out, at least with my sensetive IEMs (UE-10Pro and super.fi 5Pro). Quite possibly an impedance issue, as it might be related to the different hissing noise floor, which is significantly lower in Rockbox.

So I'd like to know two things:

1. Why is there apparently much less hiss noise floor on the 5G Ipod's headphone out with low impedance IEMs?

2. Would it be possible to ABX  two files, recorded from the iPod's headphone jack?

Maybe Febs could provide those two files???

[saratoga]

I'd agree the soundquality of the dock output from a 5G iPod can be considered equal. But I can definitely recognize a difference from the headphone out, at least with my sensetive IEMs (UE-10Pro and super.fi 5Pro). Quite possibly an impedance issue, as it might be related to the different hissing noise floor, which is significantly lower in Rockbox.

Do you mean the difference between the headphone and line out jacks?  That would be expected, the line out has no amplification and a higher impedance, so it should have a slightly lower noise floor.

If you mean a difference between firmwares, that shouldn't happen.

2. Would it be possible to ABX  two files, recorded from the iPod's headphone jack?

Yes.  Hook up the Ipod to your computer and see for yourself.  However, if you are concerned about impedance, then you MUST also hook up your headphones in parallel with the mic jack in order to provide a load for the headphone jack on the Ipod (otherwise you'll just test the unloaded noise floor, which you suggested is not the same).

[wolfB]

If you mean a difference between firmwares, that shouldn't happen.

Yes, that is exactly what I meant. To be clear, the scenario is as follows:

Low impedance high sensetive IEM on the headphone out of the 5G iPod.

Apple-FW: clearly audible hissing noise floor.
Rockbox: definitely much less hiss.

I am not sure if this is due to impedance miss match, as the hardware didn't change.

That's what is objectively reproducable. On a subjective side note I'd say the overall soundquality is improved on headphone output with Rbx (I know, I am not alone with this...see head-fi forum), IMO at least on the same level as from the dock out.
To shed some light on this subjective impression it would be great, if this could be checked in a ABX test with two files recorded from the headphone output.

I doubt I have the appropriate hardware to record the files, but I'll try.

[saratoga]

If you mean a difference between firmwares, that shouldn't happen.

Yes, that is exactly what I meant. To be clear, the scenario is as follows:

Low impedance high sensetive IEM on the headphone out of the 5G iPod.

Apple-FW: clearly audible hissing noise floor.
Rockbox: definitely much less hiss.

I am not sure if this is due to impedance miss match, as the hardware didn't change.

That definately sounds fishy.  Could you confirm it with either ABX or RMAA results?  

I doubt I have the appropriate hardware to record the files, but I'll tr

You only need a PC + audio cable, so hopefully thats not too big a problem.

Edit:  Well I suppose if you think the issue is the load induced by your headphones, then you need a headphone splitter so that you can have both the headphones and PC listening at once (or an audio cable that you were willing to cut and splice into).

[wolfB]

Thanks for the tips, saratoga.

I'd suppose that it could be confirmed with ABX.

I'll try to record two wav files, and will post detailed results here. Just give me two or three days.

Edit: I am obviously unable to record decent wav-files, as I get rather destorted files with overemphased bass only. iPod Headphone out -> LineIn of nVidia onboard soundcard, WindowsXP, Audio Recorder 3.0. All EQ-settings flat or disabled, volume on the iPod with Rbx tried at -25db, 0db, and +6 db, but no avail. I'd think a software recorder with a peek-meter would be helpfull. Could anybody point me to a freeware download, please.

 

[Febs]

Goldwave or Audacity would do it.

Also:

1.  Check your soundcard to make sure you are plugging into the line-in, and not the microphone in.

2.  If your sound card does not have separate inputs, check your soundcard's set up program to see if there is a way to set it to line-in in software.

3.  If not, use the volume mixer to turn down the input volume on the mic/line input.