Codec Efficiency Comparison Test (iPod)

[senab]

Intro

Being a bit bored today, I thought i'd do a bit of a comparison test of how efficient they are when given files with varying bitrates. I used most of the codec's supported by Rockbox, but left some out (WAV - obvious, AC/3 - fairly useless at low bitrates). And not to mention these results are only really useful for iPod's and other PortalPlayer based builds.

Method

To do this test I compiled a new 5G iPod Video build from cvs, but edited the debug_menu.c file to not display the buffer progress bars in the 'View Audio Thread' screen. This was because the progress bars would use extra cpu cycles to render, and I wanted make the cpu to be doing as much audio decoding as possible.

I then picked a track off a cd, and start encoding it with various encoders and settings (which can be seen in the table below). For a more detailed look at the files, this analysis was done here.

I then loaded my iPod with my new Rockbox build and the files in a folder. When booting Rockbox I cleared my settings, thus making sure no DSP or equalizer was on. I also changed the WPS to one without a peakmeter (boxes_320x240).

I then played back each file one by one, leaving it on the WPS screen until around 50% of the track was played. I decided to wait to 50% because by then, the audio would be filled and the boost ratio would have levelled out. I then went into the Debug menu --> View Audio Thread, and waited for the boost ratio to become level before recording the value.

Results

See here

Conclusion

After the recent optimizations to libFaad, AAC is decoding very nicely in realtime now. MP3 & Vorbis are decoding in very similar speeds across the bitrates, Vorbis has quality gains in lower bitrates though. Musepack was the most efficient lossy codec on test. I was quite suprised by just how well FLAC decoded even using q8 which is the highest compression level using libFLAC.


I'm not sure whether this test was anything good or worthwhile, but it may give you an indication of how efficient the codec's you're using are. Now of course, this test was done with a build not using any DSP's or fancy graphics, therefore if you're using these don't expect anything as low.

[bk]

I'm assuming by "iTunes" and "Nero" you mean LC-AAC, right? I don't use Nero but I read somewhere that at low bitrates it will encode in HE-AAC.

Interesting results. I would have expected libmad (MP3) to be the fastest.

[senab]

Yes LC-AAC. I put both Nero and iTunes in because Nero use's is "true" VBR, whereas iTunes' implementation of VBR is ABR, I just wanted to see if it makes any difference.  

[Davide-NYC]

Here's a stupid WPS that may be helpful: It has no alternating sub-lines, no peakmeter, no progressbar, no statusbar, etc.

I does give you the essential information you need for testing: playlist length and current position, filetype, bitrate, VBR/CBR, etc.

Let me know if this is at all useful.  

[attachment deleted by admin, too old]

[Rincewind]

Let me know if this is at all useful.  

No it's not  
(but it might be useful for other people)

When I "test" performance I usually leave the settings just the way I am normally using rockbox, "No side-effects" is an utopia anyway  

[soap]

When I "test" performance I usually leave the settings just the way I am normally using rockbox, "No side-effects" is an utopia anyway

While I understand that sentiment, give me a chance to explain why at this stage in the game it is counter-productive.

Runtime and efficiency data are most useful when they are collected in a consistent manner with the number of variables reduced and documented to the best of the tester's ability.

Look at the current status of the runtime wiki pages, for example.  http://www.rockbox.org/twiki/bin/view/Main/IpodRuntime and http://www.rockbox.org/twiki/bin/view/Main/IriverRuntime.
These pages, in their current state, (no offense to the generous people who contributed) are next to worthless when it comes to tracking Rockbox's progress in achieving longer runtimes.  Without the documentation of settings used there is no way to tell if the runtime is effected by CPU-hungry options or WPSs with lots of cycle-stealing eye-candy.  Without performing the same test with the same files on the original firmware there is no way to judge the condition of the tester's battery.

By the same token the Rockbox forums and Wiki are not exactly overflowing with runtime and efficiency data, despite the fact this would be a very easy way for non-developers to contribute.  If just 1% of the users unhappy with ipod battery life could be encouraged to...I must be smoking something.

Since there is so little data collected, I feel it is of utmost importance that the tester take that extra step to do the collection in a manner which is most useful to the study of performance as a whole, and not necessarily the most applicable to their personal usage patterns.  At the least I beg of them to collect both.

[Davide-NYC]

@ Soap: I could not, no matter how hard I tried, have stated it better.  

We need a new wiki page (CodecPerformance maybe?) which hosts the necessary sample files and a good desription of how people should run the codec performance tests and how and where to report their findings. Using a single 3 minute WAV file encoded in every supported codec and every possible bitrate should do the trick.

The best solution would be to create a WPS that reports the boost ratio directly on screen! I have no idea if this is possible or not but a true TEST.wps (rwps) would rule.

Then we need to expand BatteryRuntime to include a standard sample album and standard settings and WPSes etc and an explicit procedure on how to test and how to report back findings for battery runtime.

Then we need to solicit participation from the users. I bet IriverRuntime and IpodRuntime become full of data very quickly.

People are itching to help. Most just don't know how to get 'jiggy' with the C code. Myself included.  

[mnhnhyouh]

*sticks hand up to help with testing*

h

[Rincewind]

I didn't want to critisize your testing methods. To compare settings it is essential to minimize the influence of other factors.

My post wasn't very serious. Maybe I should have put more emoticons in

The best solution would be to create a WPS that reports the boost ratio directly on screen! I have no idea if this is possible or not but a true TEST.wps (rwps) would rule

Yes, I was thinking of that, too. I don't know if it easy to do, because these values are probably in seperate threads. If I can do it in a few hours I might try it next weekend.

[Llorean]

Why would a WPS need to show the boost ratio at all? Why can't you just leave it on the audio thread screen?

[Davide-NYC]

You're correct. Better to slightly expand the Audio Thread screen.

If the audio thread screen told you the current filetype and bitrate as well as time remaining in the current track we'd be set.

 

[Llorean]

Why is all that information important to a codec efficiency test? I would assume that if you were testing it, you'd prepare a playlist containing only the desired filetype and bitrate in advance. As for time remaining in current track, I'm not so sure I see that as useful at all for this.

I'm just saying, I'm not sure any coding actually needs to be done at all, though I could easily be missing where the information would be valuable on that screen.

[Davide-NYC]

Maybe I am misunderstanding something or there is a bug in the "View Audio Thread" screen which is causing confusion...
--> What exactly does the "track count" tell us? It is not reporting what I expect it to.

As far as reporting some file info in the View Audio Thread screen:
It would reduce human error and be much easier of the filename, the filetype and the bitrate where displayed on the View Audio Thread screen. If multiple samplerates are implemented then we should display that too (if at all possible)

Since Senab has disabled the "bufferbars" anyway which requires a patching debug_menu.c I'm figuring it's no different to the tester if we add a line on screen that reports file info.

In fact, I vote that if this change gets implemented it should be commited to CVS. It's just useful IMO.

[Davide-NYC]

Line 350 of apps/debug_menu.c insert -->

struct mp3entry* id3;
id3 = audio_current_track();
       
snprintf(buf, sizeof(buf), "filename:");
lcd_puts(0, line++, buf);
       
snprintf(buf, sizeof(buf), "%s", id3->path);
lcd_puts(0, line++, buf);

snprintf(buf, sizeof(buf), "codectype: %d", id3->codectype);
lcd_puts(0, line++, buf);
       
snprintf(buf, sizeof(buf), "bitrate: %d", id3->bitrate);
lcd_puts(0, line++, buf);

[Llorean]

I'm still not sure why that's at all relevant to a codec efficiency test. All it does it inflate the binary a wee bit more with information already available in plenty of places, and in fact, information that the tester *should* have decided on before starting the test.