Codec Efficiency Comparison Test (iPod)

[Davide-NYC]

If I'm testing 21 files I want to make sure I'm testing the file I'm supposed to be testing.
I know myself and I know I'll get distracted or miss a file or whatever. This way confusion is impossible.  I also want to minimize the WPS as much as possible in terms of CPU load. If we're to solicit the regular users (like me) in helping us test I feel it necessary to give something clear and comprehensible to look at.

Don't worry, I don't have CVS write access and I don't want it.

[Rincewind]

maybe a nice solution would be a real (unofficial, nothing in cvs of course) testing version of rockbox with dedicated testing features like flushing the audio thread, clearing buffers with button presses, triggered timers...

but this would increase cpu load a little bit. But the question is: do we want to find out what the settings with the best cpu efficiency are or do we want to find the "best" codec?

[senab]

How do you define the best decoder then?

Some people want more battery life
Some people want to fit more music on their player
Some people want both

What's best is personal to the user  

[Davide-NYC]

My intention was just to see how different codecs performed across targets at a given time.

(been busy lately, will get to the wiki page ASAP)

[soap]

The wiki entry is looking nice, but after looking at Senab's encode.bat I feel it is important to say that these are not CBR encodings - despite the implications made by both the filenames and the wiki page table.

 

[Davide-NYC]

OK here is the (very preliminary) wiki page. http://www.rockbox.org/twiki/bin/view/Main/CodecPerformanceComparison

I fixed senab's batchfile (it had a tiny bug in it) and I modified the patch to disply filename, codec, and bitrate in the 'view audio thread' screen.

Let me know what you all think. Running tests now.

[saratoga]

OK here is the (very preliminary) wiki page. http://www.rockbox.org/twiki/bin/view/Main/CodecPerformanceComparison

I fixed senab's batchfile (it had a tiny bug in it) and I modified the patch to disply filename, codec, and bitrate in the 'view audio thread' screen.

Let me know what you all think. Running tests now.

That looks fantastic.

[soap]

Regarding again the target bitrates:
What is the bitrate variation amongst the encoded files?  Are the mp3 encoded files within reasonable tolerances of their ogg counterparts?
The reason I ask is if the "128kb/s" MP3 is actually 120, and the "128kb/s" OGG is actually 132 (or vise versa) that would be more than enough to call the results into question.

[Davide-NYC]

The reason I ask is if the "128kb/s" MP3 is actually 120, and the "128kb/s" OGG is actually 132 (or vise versa) that would be more than enough to call the results into question.

Hmm. Agreed, I'll modify the batch file to encode in CBR in all possible instances.

This morning I was trying to test with an iRiver H3x0 and realized that since there is no HD activity LED on the unit a quick way to determine whether or not there was any disk activity during the boost ratio calculation would be to enable the status bar in the "view audio thread" screen. Good idea?

Of course I have no idea how to do this codewise.  

Any care to chime in? I would modify the VAT screen patch accordingly.

[bk]

Hmm. Agreed, I'll modify the batch file to encode in CBR in all possible instances.

Forcing CBR for natively VBR formats (Ogg Vorbis) would be misleading. I think the value of these benchmarks is not necessarily in the comparison but rather in the absolute boost level per target. In other words it's less useful to compare codec A to codec B, but more useful to know that codec A is 20% slower on ARM relative to ColdFire.

[soap]

Hmm. Agreed, I'll modify the batch file to encode in CBR in all possible instances.

Forcing CBR for natively VBR formats (Ogg Vorbis) would be misleading. I think the value of these benchmarks is not necessarily in the comparison but rather in the absolute boost level per target. In other words it's less useful to compare codec A to codec B, but more useful to know that codec A is 20% slower on ARM relative to ColdFire.

Forcing at least ABR is a must for this test to mean anything.  Comparing MP3 to OGG when the bitrates differ substantially is worthless.

As for the original encodings, only 192 and 256 are useful, IMHO.

[senab]

Yes this test wasn't really a test on bitrates against each other and you couldn't really do such a test. Bitrates will change on the individual sample, (ie Vorbis q6 is supposedly ~192, but gives me ~180 very often).

What you can do with the results is plot a bitrate vs boost ratio scatter graph and then create a line of best fit to compare how the bitrates between teh codec's.

[soap]

Yes this test wasn't really a test on bitrates against each other and you couldn't really do such a test. Bitrates will change on the individual sample, (ie Vorbis q6 is supposedly ~192, but gives me ~180 very often).

What you can do with the results is plot a bitrate vs boost ratio scatter graph and then create a line of best fit to compare how the bitrates between teh codec's.

Why couldn't you force ABR or CBR on the codecs that support it?  Even if the different codecs allocate bits inconsistently throughout the song, the test is the average boost percentage for the song is it not?  So as long as the average bitrates are comparable the test should provide a valid comparison.

[senab]

Well thats it, would using CBR put less of a strain on the CPU? It would be an unfair test then. Here's the type of graph I was thinking of:

[soap]

Maybe CBR would present less of a strain, but I believe most/all support ABR.

Regardless, your graph has won me over.
EDIT:
(as long as the X-axis is the actually achieved ABR, not the attempted BR)