Database and song duration

[Atom2]

Hi guys,
my first post in the forum but I have been a very happy Rockbox user (on an SD-modified iPod Video, Rockbox version 03d326fc90-230818) for quiet some time and have also been a long time lurker on the forum.

The reason for my post is something which I recently discovered and which I consider a possible bug or at least an anomaly in the database: I do have quiet a few MP3 files on my iPod and I have recently decided to initalize and use the database. All went well and while I was at it, I decided to also try adding lyrics to my songs. Searching the web, I found https://lrclib.net which claims to provide lyrics for a substantial number of songs.

The API to access lyrics requires to provide track_name, artist_name, album_name, and duration. So I decided to try my luck with one of my favourite songs - Albert Hammond" (artist_name), "The Free Electric Band" (track_name), from "The Very Best of Albert Hammond" (album_name) with an indicated track duration according to the database of 3:11 minutes which equates to "191" seconds (duration). To my surprise, the API-search returned no result.

I therefore decided to download the full (SQLite) database from https://lrclib.net and figure out locally whether the song is simply missing. It turned out, that the song is actually included in the database with artist_name, track_name and album_name matching, but a duration of 204 seconds (i.e. 3:24 minutes) as opposed to 191 seconds (i.e 3:11 minutes). Searching the FAQ of the website revealed that the API is rather sensitive to the duration and only allows a deviation of 2 seconds - which explains why the API was not able to find the song.

Further investigation for other songs (initially of Albert Hammond) revealed that the duration as shown in the database on Rockbox appears to be substantially (i.e. in all checked cases more than 2 seconds) off the expected value in the lyrics database. Crosschecking another artists (i.e. ABBA) also revealed substantial discrepancies (for matching album, artist and songs) with Rockbox always showing a shorter duration compared to other sources.

So I decided to play the Albert Hammond song through Rockbox and measure the time with a stopwatch. To my surprise it actually played for 3:24 minutes as opposed to the 3:11 minutes indicated in the iPod Rockbox database (which btw also coincides with the duration displayed on the Rockbox simulator under Windows, Version ccd612345c-18012 using the identical MP3 file). Another check through the Mp3tag programm shows a length of 3:26 minutes (which I attribute to the fact that this duration includes the 2 seconds gap between two songs). Windows Media Player also plays the song in 3:24 minutes.

So my question is: Where does the duration as displayed in the Rockbox database come from and why does it substantially deviate (i.e. being shorter in time) from all other checked sources which are:

• the duration as indicated on the album cover
• the actual play time
• the length as displayed by Mp3tag
• the time elapsed when played in Windows Media Player

Could it be that I have found a bug or is there a logical explanation for the observed discrepancy.

Many thanks,

Atom2

[7o9]

Step 1 would be to confirm the problem with a current build. Yours is just over 1 year old.

There have been some changes regarding playing time.

You can backup your current build by just making a copy if you want to be able to go back.

[Atom2]

7o9,
thanks for your suggestion which I have followed in a two step approach:

Firstly I updated to the latest dev build (i.e. 20240826), but left the database in place. There was no change, the song "The Free Electric Band" still showed up with a duration of 3:11 minutes (i.e. 191 seconds) under Database -> Artist -> Albert Hammond -> The Very Best of Albert Hammond.

In a second try, I also re-initialized the database using the newly installed dev build under Settings -> General Settings -> Database -> Initialize Now. After the re-initialization was done (it took quiet some time due to the large number of songs on my iPod), the duration for the song "The Free Electric Band" again showed up as 3:11 minutes (i.e. 191 seconds) under Database -> Artist -> Albert Hammond -> The Very Best of Albert Hammond.

So in a nutshell, the issue is still present in the latest version available for download.

Any more thoughts anybody?

Thanks, Atom2

[Frankenpod]

I didn't actually realise/remember that the DB listing tells you the length of the track (is that a relatively-new feature, or has it always done that?).

But having just checked, the timings are correct for everything I've looked at in my library.  RB shows the same lengths as windows, mp3tag, and discogs.com.

Could it be something odd about your mp3 encoder?  Or the files are getting corrupted in some way?  Are they VBR? (which sometimes seem to end up with incorrect bitrates reported - I think in the file headers - which maybe can cause confusion as to the real length of a track).

When you play them in rockbox does the time played/time remaining displayed in the WPS come out consistent with the actual length of the track or the length stated in the database listing?

[Atom2]

Frankenpod,
thanks for your post and your questions which I may reply to as follows:

I didn't actually realise/remember that the DB listing tells you the length of the track (is that a relatively-new feature, or has it always done that?).

I honestly don't know since when this feature is available. I have only started to use the database very recently, but so far very much like it to navigate my music.

Could it be something odd about your mp3 encoder?

I use the LAME MP3 encoder (see https://lame.sourceforge.io/), which I consider to be the golden standard ...

Or the files are getting corrupted in some way?

I very much doubt that this is the case and I have also checked the subset of files in question with no corruption showing; their SHA-hash value matches with the original file on my ZFS-NAS (RAID-Z2 w/ internal checksums for all data blocks). I currently assume that the issue of incorrect duration being shown occurs for all of MY songs (with varying deviations which appear to be the further apart the longer the song is). I still have to find a single song for which its correct duration is being displayed ...

Are they VBR? (which sometimes seem to end up with incorrect bitrates reported - I think in the file headers - which maybe can cause confusion as to the real length of a track).

All my MP3 files are VBR encoded using LAME's "true variable bitrate mode" with the highest quality (command line option "-V 0" - see https://svn.code.sf.net/p/lame/svn/trunk/lame/USAGE)

When you play them in rockbox does the time played/time remaining displayed in the WPS come out consistent with the actual length of the track or the length stated in the database listing?

The time played and remaining (for "The Free Electric Band") starts off with 0:00 and 3:11 respectively and changes (up/down) in line with the real time elapsed during playback. As soon as the 3:11 mark is reached, the song continues to play and the time played continues to increase up to the 3:24/3:25 mark, while the time remaining (consistently) moves into negative territory.

I hope that helps in further pinpointing down the issue.

Thanks, Atom2

[Frankenpod]

Honestly it reminds me of what happens when VBR files incorrectly report the bitrate in the header.  Ages ago I had a lot of files like that, that I had to correct with Foobar2000 (that has a specific option for repairing incorrect VBR headers), and, as far as I can remember, such files would report incorrect lengths, and behave just as you describe when played.  I mean, could be something else, but worth checking the files for wonky VBR bit rate information, using Foobar2000 or something similar.

[Atom2]

Frankenpod,
thanks for your reply and the pointer to Foobar2000. I did have a look at it and a few other programs that deal with incorrect header information. Please let me share my findings (all for Albert Hammond's "The Free Electric Band":

1.) Foobar2000 is obviously able to handle the issue by completely rewriting the whole file. After the operation the file has shrunk by 208 bytes, all of which have been taken out from the Xing header within the LAME structure which sits almost at the start of the file (in essence Foobar2000 decided that 208 NULL padding bytes are not required and removed those from the structure). The mode of operation of Foobar2000 by changing all of the file is however a no-go as this would mean transferring all data again to my iPod (as opposed to only a delta of changes for each file using rsync).

2.) I also looked at VBRfix which was also able to correct the issue. It however created a new file (and renamed the original as a backup file) which again interferes with my way of transferring files to the iPod (explanation: on my iPod all files and directories have 6-character long filenames which are created as a hex-representation of the i-node number the file/directory as stored on my NAS. This results in short filenames on the iPod while retaining a (decipherable) link to my NAS which, as an added benefit, is immune from move and rename operations of files and directories on my NAS. The creation of a new file, however, breaks that association). VBRfix on the other hand only changed a few items within the Xing frame (some of which, after investigation, did not make any sense to me at all), but left the rest alone (i.e. no deletion or insertion of bytes) resulting in identical file sizes.

3.) MP3val is able to validate a file and clearly flagged one issue: For all affected files tested, it complained about incorrect MPEG data size information - specifically for "The Free Electric Band" the message was "Wrong number of MPEG data bytes specified in Xing header (7743338 instead of 7089765)". This little tool was also able to correct the problem, but unfortunately also did not work in-line but instead again created a new file - with problems identical to the ones I had with VBRfix (see above).

4.) In essence, those tools still were a good starting point and provided me with an excellent baseline to investigate further. Armed with a hex-editor I tried to figure out what was required to rectify the issue, where to store the required updated information and where to get the correct number of MPEG data bytes from. It quickly turned out, that there are two fields in the Xing frame which had the data size: One pretty much at the start of the Xing frame in a field commonly referred to as bytes (see byte 48-51 on http://www.multiweb.cz/twoinches/mp3inside.htm, section VBR file structure) and another one in the LAME structure within the Xing frame named music length (see section named bytes $B8-$BB on http://gabriel.mp3-tech.org/mp3infotag.html).
The second of these fields is, according to the latter website, the exact length in bytes of the mp3 file originally made by LAME. It appears to also be the size that all three tools above did change the other (first) field with the wrong number of bytes to. I neither know where the wrong file size comes from in the first field nor do I know why Rockbox obviously uses the first entry and not the second entry given that LAME should best know what the exact audio size was when the mp3 was created. I also do not understand, why having a smaller number of bytes in the size fields results in longer playtimes for the song as displayed by Rockbox. To me this is counter-intuitive: All other things equal, the length of a song should have a direct relationship to the size of the music data.

5.) In the end I decided to write some C-code that does exactly what I want: Get the integer value (for the audio size) from the LAME structure within the Xing frame and copy its value in the space occupied by the length information near the start of the Xing header. Unfortunately that approach turned out to be more complex than I anticipated (to some degree because my coding-level was somehow rusty, but mainly due to issues with little-endian vs big-endian issues, unexpected half byte swaps when accessing data as C bitfields which I decided to use together with unions and structures to model the header and also sanity checks took some considerable amount of time), but in the end I seem to have a working piece of code that does what I want - and is pretty fast as it only reads 10 bytes at the start of the file plus another 417 bytes further back for the Xing frame which is then subsequently modified and written back in place. I'll do some more testing and then let it run on my currently collection of around 17.000 files later.

Just out of curiosity, I have also added a check routine that verifies whether the two audio-size/length fields are in line or if they deviate one from another and thus the first field requires updating. When I ran this routine on my collection, it turned out that only 11 of my odd 17.000 files have the same number in both fields. And for those 11 files, the time is correct in the iPod database.

While the problems appear to be solved (or at least solvable), it still escapes me where the wrong data originates from. I also do not understand and I'd still be interested to have somebody in the know to comment, why (so far only) Rockbox thinks the duration of the song is less than it actually is while all other players tested (including Foobar2000) show the correct time.

Thanks, Atom2

[Frankenpod]

Wow, that's a comprehensive analysis.

I don't know how the VBR headers got messed up for the files I had - something to do with the encoder I used to rip the cds - they were all ripped many years ago and can't remember what software I used, but it seems it didn't write the headers correctly.

  I assume it must be a common problem, though, hence Foobar and vbrfix have specific functions to correct it.  It seems something very specific to VBR files.  My experience, which makes intuitive sense, was that the files that were wonky reported bit-rates higher than they actually had, and durations shorter than they actually had.  I assume those two factors are related in some way so the size in kB comes out right.

I really don't know, though, why the problem would show up with RB but not other software (IIRC the problem was present on my files in players other than RB). 

[rockbox_dev123]

on my iPod all files and directories have 6-character long filenames which are created as a hex-representation of the i-node number the file/directory as stored on my NAS.

This makes you completely dependent on the rockbox database and seems a little nutty to me.

You might get some value out of this: https://web.archive.org/web/20240305070623/https://www.pkrc.net/detect-inode-moves.html

If you change even one byte in a moved file (imagine retagging your music library with respective changes of file/directory names), everything will still be transferred over network.

It’s a small Python script that will dump inodes for every file and directory within a directory tree to a text file, you then do your big changes and after you run it again, it will produce a human-readable Python (or shell) script which you can run on the remote machine to replicate the changes. If there are any remaining changes besides renaming and moving, you may finish them manually or using Unison, rsync or similar.

[Atom2]

Hi rockbox_dev123

This makes you completely dependent on the rockbox database and seems a little nutty to me.

For my use case of Rockboxs and music in general this approach works very well and I disagree that it "seems a little nutty" - but everybody is entitled to his/her own opinion. It is also worth mentioning, that the database on Rockbox opens up possibilities which file name based searching within Rockbox's file system simply does not allow - e.g. being able to easily find a song from an artist published in a compilation album of various artists. Clearly this requires correct tagging of the songs. And finally shorter file- and directory names also result in a smaller database, thus reducing RAM usage (if kept in RAM).

Furthermore, on my NAS I anyways use a dual approach: The rip-process creates files in (sub)folders "Artist/Album/Song" (w/ Artist, Album, and Song named accordingly). A small (bash) script later creates (respectively adds for new songs ripped or directories created through ripping) hard links for mp3 files and cover art into a directory structure which is used to transfer the files over to my Rockbox iPod. This hard-linking also allows me to easily segregate flac from mp3 files (I rip both formats and they are stored side by side in the same directory structure under "Artist/Album" as "Song", one with an .mp3 extension, the other with a .flac extension. By hard-linking those to different base directories named "!mp3" and "!flac" they are transparently segregated; cover art clearly ends up being hard-linked underneath both base directories).

You might get some value out of this: https://web.archive.org/web/20240305070623/https://www.pkrc.net/detect-inode-moves.html

If you change even one byte in a moved file (imagine retagging your music library with respective changes of file/directory names), everything will still be transferred over network.

I again tend to disagree here: rsync understands an option named '--inplace' resulting in files not being moved over the network but rather modified in-place on the other side. The option '--inplace' combined with '--no-whole-file' also works for local file synchronization (a USB mounted Rockbox iPod is considered a local directory providing local files) by only transferring/overwriting differing bytes from source to target, thus minimizing writes on flash storage and usually also speeding up the process (reading on flash storage is usually faster than writing).
Clearly this process works best, if the file size stays the same and only certain bytes are changed, but nothing is "added" (i.e. modification of the audio-size field, but even adding additional tags works, as long as those new entries only fill up - in other words: overwrite - spare space previously padded with NULL bytes).

From the manual page of rsync:

--no-whole-file, --no-W
              Disable whole-file updating when it is enabled by default
              for a local transfer.  This usually slows rsync down, but
              it can be useful if you are trying to minimize the writes
              to the destination file (if combined with --inplace) or
              for testing the checksum-based update algorithm.
--inplace
              This option changes how rsync transfers a file when its
              data needs to be updated: instead of the default method of
              creating a new copy of the file and moving it into place
              when it is complete, rsync instead writes the updated data
              directly to the destination file.

Please also see https://superuser.com/a/674634.

Nevertheless, thanks for joining the discussion and adding further information, Atom2

[Atom2]

sorry - double post -

[Atom2]

I have investigated some more and the issue of the incorrect duration within the Rockbox database more and more points to an uncommon/strange calculation within the Rockbox code base. From my investigations, I at least do now understand how the other players calculate their displayed song lengths - which is usually correct or at least within 1 to 2 seconds of the duration indicated on the album cover (the difference probably resulting from the silence/gap between any two songs).

Instrumental in getting an understanding about the calculation was https://www.codeproject.com/Articles/8295/MPEG-Audio-Frame-Header#VBRHeaders which shows a formula to calculate the duration of a VBR-encoded songs using the following parameters (which, interestingly enough, neither include the audio data size nor the average bit rate resulting from encoding the song):

• Number of Frames - is stored within the MP3 file through the encoder
• Samples per Frame - constant for mp3 files (which is MPEG-1 Layer III) with a value of 1152 (see the link above under section "2.1 MPEG Audio Frame Header"
• Sampling Rate - either 32000, 48000 or - for CDs - 44100 and encoded through a 2 bit sampling rate index of 00 (again, see the link above)

As per the list of parameters above, all of these values are readily available from data stored within the encoded mp3 file and the calculation then goes as follows:

Duration = Number of Frames * Samples Per Frame / Sampling Rate

For Albert Hammond's "The Free Electric Band", which has 7876 frames (stored within the MP3 file), the calculation is "7876 * 1152 / 44100" which results in 205,7 seconds or 3:25 minutes as opposed to the 3:11 minutes Rockbox comes up with. Strangely enough, Rockbox, however, seems to rely on one of the audio data size values (two differing values are also stored in the MP3 file) somehow because once this value is adjusted accordingly, the play time is corrected as well.

So I'd still be interested in understanding how Rockbox actually calculates the duration of a song and why it does not use the formula (most) other players seem to use. Probably Rockbox developer @saratoga may be able to shed some light on this as, already some time ago, he has been active in a thread on another forum (https://hydrogenaud.io/index.php/topic,114292.0.html) which, during my research, pointed me to the website linked above. The issue in that thread was an incorrect bit rate displayed by certain players for VBR encoded files which @Frankenpod suspected as being the culprit for the wrong duration being displayed in Rockbox.

Thanks, Atom2