Codec Efficiency Comparison Test (iPod)

[Llorean]

Hmm...

There is ONE small problem about all this: It's hard/worthless to compare between say, Coldfire and ARM still.

Because of differences both in processor architecture, unboosted and boosted speed, and operating system overhead, boost ratio really wouldn't meant that much when comparing the two.

[soap]

I think the plan was all along to collect two different data-sets and never the two shall meet.

[senab]

It's the achieved ABR

Yes, ABR would probably be the better option, but most codec's are optimizied for VBR output. Vorbis has no true CBR output, neither does Musepack or Nero; even the iTunes CBR bitrate can fluctuate.

[Llorean]

Another problem (native to the coldfire on H100 at least) then, is that MP3 in MANY cases can exist unboosted.

It may be necessary to clock down the processor further before starting testing, so that you don't have worthless data points (in my mind 0% boost is without value since you don't know if it's exactly 0%, or less, or more).

The real BEST way to test codec efficiency is this I think:

Create a transcoder to WAV. Then, time how long the transcode takes (while boosted the whole time) for files of various bitrates/qualities.

Do the transcode without yielding so that 100% of CPU time goes only to it. Don't even bother saving the data, just discard.

[soap]

That methodology seems sound.

[bk]

There is ONE small problem about all this: It's hard/worthless to compare between say, Coldfire and ARM still.

Because of differences both in processor architecture, unboosted and boosted speed, and operating system overhead, boost ratio really wouldn't meant that much when comparing the two.

Not true. If a codec is running at 90% boost on one arch and 60% boost on another, then clearly the optimization effort would be best directed towards the first architecture. "Operating system overhead" is meaningless since Rockbox is running on both.

[Llorean]

You clearly don't understand the situation.

On on architecture, it's boosting from 30 to 75. On another it's boosting from I think 45 to 124. So, the 45 one may boost 0% of the time simply because the codec runs fullspeed at 45, while it boosts some of the time because it doesn't run full speed at 30.

It is an unequal comparision of how optimized the codecs are simply because the processors are in seperate conditions.

Operating System Overhead IS DIFFERENT between different hardware. Because Rockbox is being compiled into ARM assembly vs M68K assembly operations take different amounts of time to complete. This means that basic code like User Input handling can be less or more efficient on one architecture than another. Then when you get into the code, because one has different inputs than the other, this introduces further differences in operating overhead. Take into account that each screen requires a different amount of time to update, and it will always be updating as long as you yield to the UI thread, and you get even *more* differences in operating overhead because the OS on one hardware has to spend more time drawing than on the other.

There are VAST differences in how Rockbox itself performs on different hardwares, completely independent of codec performance.

For examples of this, try scrolling in long lists on an H300 vs an H100, or on an iPod Nano vs an iPod Photo vs an iPod Video.


Using a transcoder at full boost that does not yield you accomplish a few things:
1) You prevent any other code from executing, insuring that you're ONLY timing the codec itself.
2) You are running the codecs using the full power of the processor without ANY questionable overhead, I believe, which then means that you have MP3@128 on ARM7 @ 75mhz, vs MP3@128 on M68K @ 124mhz. Then you at least only have three real variables: The codec itself, and the maximum processor speed, and the differences architecture makes (A 75mhz ARM7tdmi is not the same actual speed as a 75mh M68K Coldfire).

As far as I'm aware, there's a ridiculous amount of things that can cause a variance in the results.

If it's running at 90% boost on a processor that only goes up to 60mhz and runs at 25mhz idle, vs 60% boost on one that runs at 50mhz idle and 200mhz boosted, its pretty clear that it's running less efficiently on the latter, and this case could reasonably come up. The only proper solution is to figure out how efficiently it runs relative to the architecture itself, which means you need to be able to establish a performance value that is independent of actual processor speed and any hardware you can possible remove from the equation.

[bk]

You clearly don't understand the situation.

Do me the favor of not presuming what I do or do not know.

On on architecture, it's boosting from 30 to 75. On another it's boosting from I think 45 to 124. So, the 45 one may boost 0% of the time simply because the codec runs fullspeed at 45, while it boosts some of the time because it doesn't run full speed at 30.

a) On different architectures cycles are not 100% equivalent; b) if the goal is to increase performance of various codecs then boost ratio is an adequate metric to use, regardless of hardware differences.

It is an unequal comparision of how optimized the codecs are simply because the processors are in seperate conditions.

Somewhat true but irrelevant. If codec A has 10% boost on ARM and 75% boost on ColdFire then optimization effort on ARM is not as worthwhile as effort spent on ColdFire for that codec.

Operating System Overhead IS DIFFERENT between different hardware. Because Rockbox is being compiled into ARM assembly vs M68K assembly operations take different amounts of time to complete. This means that basic code like User Input handling can be less or more efficient on one architecture than another. Then when you get into the code, because one has different inputs than the other, this introduces further differences in operating overhead. Take into account that each screen requires a different amount of time to update, and it will always be updating as long as you yield to the UI thread, and you get even *more* differences in operating overhead because the OS on one hardware has to spend more time drawing than on the other.

Also do me the favor of not explaining the process of compilation as if I was a child. Rockbox is the operating environment on all architectures: if there are threading inefficiencies (for example) on one target affecting codec performance then it would be exposed through these tests (indirectly) and could be addressed. This is the entire point of performance benchmarking.

[Llorean]

In response to your "Cycles are not 100% equivalent" you may have noticed in many places I mentioned that very point.

You seem to think that all that matters is its boost ratio. Which is completely pointless since the processors run at different speeds, the ARM core we currently have being half the speed of the coldfire.

Your theory is that even if a codec is running *faster* on one architecture than on another, if the processor is slower and thusly it must boost more, concentration should go there. Which is silly to an extent, because the codec is actually *more* efficient in that situation, and optimization efforts could very well be wasted time or diminishing returns. It's better to know an absolute value of efficiency. You can still concentrate more on the slower processors if the speed/efficiency ratio means that processor will be boosting more, but it also gives you a benchmark upon which to base  overall speed for that architecture. Particularly useful in considering future targets. For example, if you're considering a slower ARM based target, you can have a better idea what may or may not be feasible on it.


And the reason I suggested you don't understand what's going on, is because you said ""Operating system overhead" is meaningless since Rockbox is running on both." which is what I responded to with that. It is clearly an untrue statement, as this thread is relating to CODEC EFFICIENCY, which means the tests should be related to the Codecs themselves, not Rockbox performance on a given system. It does differ, and hardware (particularly screens) interfere GREATLY and cannot necessarily be improved. This is not a fact that can be simply washed away with the word "irrelevant."

Otherwise we can simply say "All codec optimization efforts should happen on the 5G iPod(or perhaps 3G iPod)" because simply put it has the single worst playback performance of any current system other than the 3G iPod with its broken cache.

Edit: As a side note, please don't talk back to me about explaining compilation. The VAST majority of users here are not very aware of such topics, and I have no way of knowing whether you are or aren't. It is not speaking down to you as if you are a child, as VERY few childs understand that concept. It's speaking down to you as if you were "average" which is often not considered speaking down to someone at all, simply speaking to someone who may not be aware of the details of a technical concept. If you cannot discuss this idea without getting offended, I suggest you step away from it for a few hours and coming back when you have cooled down, as in no way is ANYTHING I say here meant to belittle you. Just state my perception on the topic. As far as I'm concerned, I said nothing untrue up to and including the fact that your statements demonstrated a lack of knowledge about the causes of differing performances across hardware and the relevancy of operating system overhead on the topic of "Codec Efficiency Comparison".

[saratoga]

There is ONE small problem about all this: It's hard/worthless to compare between say, Coldfire and ARM still.

Because of differences both in processor architecture, unboosted and boosted speed, and operating system overhead, boost ratio really wouldn't meant that much when comparing the two.

Not true. If a codec is running at 90% boost on one arch and 60% boost on another, then clearly the optimization effort would be best directed towards the first architecture. "Operating system overhead" is meaningless since Rockbox is running on both.

I think what Llorean meant was that boost doesn't tell you which is faster, not that the numbers themselves were useless.

On on architecture, it's boosting from 30 to 75. On another it's boosting from I think 45 to 124. So, the 45 one may boost 0% of the time simply because the codec runs fullspeed at 45, while it boosts some of the time because it doesn't run full speed at 30.

It is an unequal comparision of how optimized the codecs are simply because the processors are in seperate conditions.

I think bk just means comparing the relative boost ratios tells you which platform needs optimization most.  I'm not really sure why we would care about that, but its a valid point.  I agree with you that it doesn't really mean anything though.

Somewhat true but irrelevant. If codec A has 10% boost on ARM and 75% boost on ColdFire then optimization effort on ARM is not as worthwhile as effort spent on ColdFire for that codec.

I don't see how you can conclude this.  Given the differences in ISA, power consumption and battery capacity, its entirely possible that they're equally worthwhile.  For instance, Coldfire could be highly optimized, but poorly suited for the task, while ARM could be poorly optimized, but well suited.  

At any rate, since the developers working on each platform are different people, its not very relevent.  Knowing that X needs optimization more then Y doesn't help if theres a fixed group of people who work on X and a seperate group that only work on Y.

[Llorean]

As far as I'm concerned testing boost only gives you relevant information over the normal playback condition, and comparing it boost between codecs on the same platform is fine, but between platforms destroys the value of the data. If all the codecs require more boosting on one system than the other, is it that all the codecs are still poorly optimized, or is it that the playback code requires effort?

Knowing that an MP3 decodes at 300% speed on ARM7 @ 75mhz, and 350% speed on M68K @ 124mhz is much more relevant information to overall codec efficiency on those platforms.

If this were a Playback Efficiency thread then I would gladly admit that operating system concerns would be relevant to the topic (and they are relevant to users) but either way they should still be measured entirely seperately of the codec (playback system should probably be measured with WAV files).

[Davide-NYC]

In trying to refine the testing process:

CBR, ABR or VBR? I think whatever we choose should be the most "default" coding option. I.E. the encoding options used the  majority of the time. What is the batchfile going to tell the encoders to do exactly? Do we have concensus on this? My vote is use the encoder defaults because presumably that's what most file are encoded with.

OK, so the Coldfire and ARM data sets should not be merged. Do we need to separate the different PortalPlayer chips? PP5002, PP5020, etc?

Coldfire Set:
iriver iHP-100    
iriver H120    
iriver H140    
iriver H320    
iriver H340    
iAudio X5

ARM Set:
iriver H10 5/6GB    
iriver H10 20GB
iPod 3G    
iPod 4G
iPod mini (1G & 2G)
iPod Color
iPod Nano
iPod Video

  What about the statusbar in "VAT" screen for HD activity? Many targets do not have an HD activity LED. It's nice to be able to see if disk access is interfering with the CPU boost ratio. Please post some code for me so that I may expand the debug_menu.c patch further.

About the transcoder to WAV plugin:
This is a great idea. The plugin could output performance data to a logfile. Just like the battery_bench plugin. This would be MUCH EASIER for the user to do.

Once the Codec_Bench plugin is written (hint hint) this entire thread becomes moot right? We would just tell users to download and encode the test files and run the plugin until it is finished. Right?

[Llorean]

A more useful feature would be a transcoding interface that just happens to have a "benchmark" option somewhere that causes it not to output a file.


For your list of players, the primary things that will make a difference are: Screen and Processor. The available RAM should be entirely irrelevant to an actual codec test as long as you're making sure the whole file gets buffered, so the H110, H115 and H120 should perform identically to the H140, and the same is true between 320 and 340. The iPods are all different though because of either different screens or different PortalPlayer chipsets.

[Davide-NYC]

A more useful feature would be a transcoding interface that just happens to have a "benchmark" option somewhere that causes it not to output a file.

What I meant by output was just a logfile that listed decoding times and whatever else was relevant to the test...

Example:

Target and Build Date.

flac_5.flac   --> time.
flac_8.flac   --> time.
lame_096.mp3  --> time.
lame_128.mp3  --> time.
lame_192.mp3  --> time.
lame_256.mp3  --> time.
lame_320.mp3  --> time.
mpc_096.mpc   --> time.
mpc_128.mpc   --> time.
mpc_170.mpc   --> time.
mpc_224.mpc   --> time.
mpc_300.mpc   --> time.
mpc_350.mpc   --> time.
nero_096.m4a  --> time.
nero_128.m4a  --> time.
nero_192.m4a  --> time.
nero_256.m4a  --> time.
nero_320.m4a  --> time.
nero_400.m4a  --> time.
vorbis_096.ogg--> time.
vorbis_128.ogg--> time.
vorbis_192.ogg--> time.
vorbis_256.ogg--> time.
vorbis_350.ogg--> time.
vorbis_500.ogg--> time.
wv_fastx3.wv  --> time.
wv_normx4.wv  --> time.


That would rule and take all of the human error out. It would also be easy for the users to do.

Although, now that I think about it a transcoder that does "non-realtime-format-->WAV" would be a great tool for playing back files that are not playable on Rockbox (yet) while not in front of a computer. Hmmm!  

[Llorean]

No, I understand what you meant by output. Rather, what I meant was that instead of a codec benchmark plugin, it'd be more useful to spend the time working on a full transcoding interface, so you can take *any* supported file, and have it decoded to WAV and then encoded to the format of your choice.

Then, once that exists, a benchmark option could be added to it that fits most of the needs above, and as a bonus because it's a full transcode interface, could benchmark both encoding and decoding codecs.

By the "instead of outputting" bit, I just meant that when using the interface to benchmark, instead of saving a file it could log it or output time results to the screen.