Codec Efficiency Comparison Test (iPod)

[Davide-NYC]

Flyspray feture request? (plugin section)

I think this is a tremendous idea.

Especially since non-realtime encoded files could then be decoded and at least played without being in front of a computer.

Me likee.

[Llorean]

http://www.rockbox.org/tracker/task/6152 This read alright?

[Davide-NYC]

Perfect! If we standardize the test files we'll get some useable results back right? Maybe this could be bundled with battery_bench and it could be a two-in one test?

Something like this?

Attention all users that want to help:
Turn on the TSR plugin. (Transcode and Battery Bench)
Play the "test file folder" in a loop untill the player shuts off.
Upload the logfile(s) somewhere.

Sounds easy enough for everyone!

What am I saying? I must be smoking too...  

[bk]

I think bk just means comparing the relative boost ratios tells you which platform needs optimization most.  I'm not really sure why we would care about that, but its a valid point.  I agree with you that it doesn't really mean anything though.

Assuming all codecs run realtime for all valid bitrates, the only reason to optimize further is to improve battery life. Otherwise all codecs could run at 100% boost for all we care. In that context be useful to know which codecs drain battery the most, and see if it is possible to improve it.

I personally don't see the point in seeing which codec is 'best' (the original intent of these tests), performance testing is only useful if it can be used to improve the codebase.

I don't see how you can conclude this.  Given the differences in ISA, power consumption and battery capacity, its entirely possible that they're equally worthwhile.  For instance, Coldfire could be highly optimized, but poorly suited for the task, while ARM could be poorly optimized, but well suited.  

I don't understand what you mean by 'well suited'. These are all general purpose embedded processors, if code is well optimized for ColdFire it will run fast and the tests will show that. Likewise for ARM, etc.

At any rate, since the developers working on each platform are different people, its not very relevent.  Knowing that X needs optimization more then Y doesn't help if theres a fixed group of people who work on X and a seperate group that only work on Y.

Agreed, work can only be done when there are people able and willing to spend time working on the various targets.

[soap]

I personally don't see the point in seeing which codec is 'best' (the original intent of these tests), performance testing is only useful if it can be used to improve the codebase.

Regardless of who said what, the original point of the test was simply to collect data.  You can't have too much objective data.  (let's not labour that point (for we could I guess)) Tests measure, nothing more.  "Which codec is best" is a judgment, not the subject of a test.

[saratoga]

I don't see how you can conclude this.  Given the differences in ISA, power consumption and battery capacity, its entirely possible that they're equally worthwhile.  For instance, Coldfire could be highly optimized, but poorly suited for the task, while ARM could be poorly optimized, but well suited.  

I don't understand what you mean by 'well suited'. These are all general purpose embedded processors, if code is well optimized for ColdFire it will run fast and the tests will show that. Likewise for ARM, etc.

Theres no reason well optimized code will run fast.  The problem may simply difficult or poorly suited for the hardware.  For instance, code that depends on coldfire's MAC may never run well on PP CPUs that have to do seperate multiply and add operations.

[Davide-NYC]

Now that the test_codec plugin exists we can now revisit this topic.

The test plugin, when using the "Speed Test Folder" option outputs a very simple log file. If we could append the results in a Twiki friendly format to the end of the same file we'd be on our way to getting many users to submit results for all targets. (I'm not sure who submitted the plugin to begin with)

• We need to expand the encoder batchfile to include WMA and (maybe) APE.
• We may want to choose a more dynamic, less noisy sample file.
• If the test_codec output could be modified to be a bit more Twiki friendly that would rule.
  
  

See here if you're not sure what I'm talking about.