Database Building and Album Track Number Sorting Issue

[bennor]

Installed Rockbox current build version ee1f54c-120226 on to 4GB Clip+ with 16GB microSD card. I am having trouble with multi CD audiobooks with more than 10 media files not listing in numerical order when viewing them via the Database/Album/<media collection name>. With the Sansa firmware these files would order properly. It would appear the "track" field and or the "disc number" fields are being incorrectly ordered in the database album page. The files are sorted 1-10, then 100-109, then 11 then 110-119, and so on. I've search through the forum for this issue and while there are some posts, http://forums.rockbox.org/index.php/topic,8611.0.html, http://forums.rockbox.org/index.php/topic,22853.0.html, that are similar to the issue I am experiencing, there doesn't seem to be a solution to the issue. I've tried to use the tagnavi_custom.config file with either of the following lines of code which has no affect on the file/track order.

%format "fmt_title" "%03d. %s" tracknum title ? tracknum > "0"
or
%format "fmt_title" "%02d.%03d. %s" discnum tracknum title ? discnum > "0"

Is there a special trick to get the Database/Album/<collection name> page to order lists of songs/audio book files that number more than 10 files correctly? Or do the ID3 tags for the files need to be changed and if so what entries are used by the Database to sort the files/tracks correctly?

The other issue I have is when initializing the database it hangs with the message "Building database... 6255 found(OFF to return)". It appears the database has trouble with the media (MP3) files on the microSD card since with the card removed I can initialize the database. Is there a limit to the number of files the database can sort through? Can the ID3 tags cause the database sorting to hang?

[saratoga]

The hang sounds like a bad file crashing the parser.  If you could find and upload the file, it would be helpful to us in fixing that bug.  If you go into System -> Debug there is a metadata log. This will make a txt file in the root of the player with a list of each file the database parsed.  If it crashes, the last item listed there is probably the one that crashed it.

As for sorting, I think changing the tagnavi file should fix it, and that we should probably update the default order to handle this.  That said I don't use the database, so I'm not too sure what to change.

[bennor]

Did the System -> Debug -> Metadata log step and while that yielded a rather large log file the last file listed each time was the last file in different sub folders on the microSD card. I'm probably going to have to experiment by moving one audiobook folder over at a time, or in groups, to see which one might be causing the trouble. I have some 5,620 files contained in 77 folders in the Audiobook folder on the microSD card with 192 files in 21 folders in the Music folder, and 6 files in the Podcast folder on the Sansa 4GB Clip+ player.

I appear to have fixed the problem with the sorting when having disc numbers in double digits and track numbers that span three digits by using a tagnavi_custom.config file. On a side note, there doesn't appear to be clear directions on editing this file on the Rockbox site, it took reading several other posts in the forum, and their trials and tribulations with editing the tagnavi_custom.config file along with some trial and error to figure out what to edit in my tagnavi_custom.config file. Here is the steps and code changes that appears to work for me.

Copied the file tagnavi.config and renamed it tagnavi_custom.config. Used Notepad (in Windows) to edit the tagnavi_custom.config file and commented out the line %include "/.rockbox/tagnavi_custom.config" with a "#". Or one can delete this line.

Next, and this is a key point to this fix, did a search and replace to renamed the entry "fmt_title" to "fmt_title_custom". Note: Failure to rename this entry in the tagnavi_custom.config causes updated code for the disc and track numbers to not work (at least for me). Next, since both the disc number (discnum) and track number (tracknum) values were causing the sorting problems, edited the following lines to change the two values. Note: I also updated, and renamed, the entry fmt_best_tracks to fmt_best_tracks_custom since tracknum is used in that entry.
From:

Code: [Select]

%format "fmt_title"       "%d.%02d. %s - %02d:%02d" discnum tracknum title Lm Ls ? discnum > "0"

%format "fmt_title"       "%02d. %s - %02d:%02d" tracknum title Lm Ls ? tracknum > "0"

%format "fmt_best_tracks" "%02d. %s (%3d)" tracknum title autoscore

To:

Code: [Select]

%format "fmt_title_custom"       "%02d.%03d. %s - %02d:%02d" discnum tracknum title Lm Ls ? discnum > "0"

%format "fmt_title_custom"       "%03d. %s - %02d:%02d" tracknum title Lm Ls ? tracknum > "0"

%format "fmt_best_tracks_custom" "%03d. %s (%3d)" tracknum title autoscore

With these changes media with two digit disc numbers and three digit track numbers should now sort correctly in Rockbox Database --> Album --> <content name> page.

The only downside to this edit to fix the sorting is all track numbers will now be displayed as 3 digits rather than 2.

Hopefully this information will help others who experience the track and disc number sorting issues that I experienced.

[bennor]

For some reason more than half of my audio book folders contain at least one file that causes the Rockbox database generation to fail. Many of the audiobook files were ripped from audio cd's with iTunes, version 10.5.3.3.

I've included a link to one MP3 file in zipped format that breaks the database generation all by itself. The audio file appears to play without issue.

http://tinyurl.com/7eun86a

[saratoga]

I've included a link to one MP3 file in zipped format that breaks the database generation all by itself. The audio file appears to play without issue.

http://tinyurl.com/7eun86a

That file is built into the database without issue on my Clip+ and in the UI simulator.  Not sure why its causing issues for your player.  Have you tried checking the file system for errors?  Perhaps your copy of the file was corrupted. 

[bennor]

I have checked both the Clip+ drive and the microSD card for errors and none were found. I even copied the file to the Clip+ internal memory and with the Sandisk 16GB microSD card in the player the database fails to build. When the file is deleted the database builds properly and finds all content on the Clip and the memory card. But, the strange thing is if memory card isn't present in the device while the file resides in the Clip+ memory the database reads the file. For some reason the database doesn't build when the file and the microSD card are both present in the Clip+.

I'll have to see if I can lay my hands on another microSD card and test it out to see if it also exhibits this strange problem.

[bennor]

An update. Bought a brand new 16 GB Sandisk (mobile ultra sdhc I) microSD card yesterday, loaded 33 folders containing audiobook files (approximately 13 GB's of files) to it and the Rockbox database appeared to read the files correctly and operate correctly. However this morning when loading a one more audiobook folder the database failed to generate properly. When the Clip+ is rebooted the Rockbox indicates its "committing database x/9" then goes to the main screen. When Database is then selected it indicates the database is not ready, initialize now yes/no.

Reformatting the microsd card and randomly loading just a few audiobook folders, the database generation appears to work. Sometimes the database appears to properly generate yet shows the individual media length to be "00.00". Then if the Clip+ is rebooted I get the "committing database" message above along with database is not ready, initialize now message when selecting Database from the menu.

Does the Roxbox database feature have the same file number limitation (8,000 or less) as the Sansa firmware on the Clip+? Using the stock Sansa firmware the Clip+ stopped recognizing files (from the same collection of audiobook files) at around 5,000 files. Beyond that no additional media would be listed, displayed, or played by the Sansa Clip+ firmware. It appears something similar is happening with the Rockbox firmware, either that or there is some kind of a bug in the Clip+ when using larger capacity microSD cards that can store large amounts of individual files.

Currently falling back to using the Rockbox File -> microSD to search through all of the content on the microSD card to play the media content.

[bennor]

Another update. After some fiddling around with the Clip+ it seems when the database crashes it leaves the file database_tmp.tcd file which is 2,296 kb in size in the .rockbox directory on my Clip+ that has, on a 16GB Sansa microSD card, 11.4GB of media (audiobook MP3 files) in 5514 individual files in 69 folders.

After much experimentation I think I found a way around this database building crash issue. Here is my workaround to get the database to build without crashing that's worked twice now on my Sansa Clip+.

• Initialize the database by selecting Initialize Now from the Settings/General Settings/Database menu and let it crash (usually at Committing Database 7/9, then Building database 6154 is displayed). Do not reboot or power off the Clip+.
• Then connect Clip to PC and delete the .TCD files including the database_tmp.tcd file which is 2,296 kb in size in the .rockbox directory.
• Then disconnect the Clip+ from the PC, do not reboot or power off the Clip+, and enable Gather Runtime Data from the Settings/General Settings/Database menu.
• Then select Initialize Now from the Settings/General Settings/Database menu, the database will start to build and after a minute or so the Clip+ crashes with the message Undefined Instruction on the screen.
• Press any button (I pressed the play/pause button) and the Clip+ will reboot.
• After rebooting, the message Committing Database 1/9 is displayed and cylces to 9/9, then the Clip+ loads and the database is populated and accessable.

[blr_p]

I encountered a similar failure of database rebuild when i installed a 64GB micro SD card. The initialise never got beyond the 4th stage. So everytime i booted up i'd get that database building message....

Pulled the 64GB out and stick the older 32 GB one in and everything worked fine.

Ran the metalogging on the 64GB and when looking at the file its gone through all the files on the 64GB card. Cannot tell where the problem was or even if there was a bad file.

My work around to is was to pull the card out, let the db initialise on the onboard memory. Switch the clip+ off, stick the 64GB card in and never update the db at all. Its worked fine ever since. Everything on the 64GB is accessible via the files option.

[thesebastian]

The hang sounds like a bad file crashing the parser.  If you could find and upload the file, it would be helpful to us in fixing that bug.  If you go into System -> Debug there is a metadata log. This will make a txt file in the root of the player with a list of each file the database parsed.  If it crashes, the last item listed there is probably the one that crashed it.

As for sorting, I think changing the tagnavi file should fix it, and that we should probably update the default order to handle this.  That said I don't use the database, so I'm not too sure what to change.

THANKS, i just solved this issue thanks to your post. I was trying to link 50GB of Music.

The problem was 1 mp3 of "Steve Vai".