Database file format, sorting, and track display formatting

[wackymorningdj]

I've seen numerous requests for user-defined database sorting order (using sort tags in present in individual files), and I've also been frustrated that, for instance, The Beatles shows up under "T" instead of "B".  I looked at the tagcache source a while back thinking of how I might be able to contribute, but got a little lost.  Today I thought that I might just be able to take a whack at modifying the database files themselves (i.e. .rockbox/database_0.tcd, etc.) by hand.  I read up on the file format (http://www.rockbox.org/wiki/TagcacheDBFormat) and played around with the python scripts on that page to get a feel for what I was up for.

I went ahead and changed "The Beatles" in database_0.tcd to "Beatles, The" hoping it would solve the problem.  I booted up rockbox, looked at my database, and was frustrated to find that "Beatles, The" did not appear in the B's . . . but then I scrolled to the T's, and lo and behold, I found "Beatles, The" where "The Beatles" used to be.

So here is my question: Do the database_[0-8].tcd files preserve order when they are displayed in the database menus?

I understand that a solution wouldn't be as simple as moving a given entry in the database cache files (since all songs are linked to their metadata via byte offsets in the database_idx.tcd file).  But if I were to programmatically rewrite all of the database files (in the process reordering the artists), is there a known reason that (a) something would break, or (b) the database files would just get overwritten later, making this futile?


On a related note, I've also seen requests for using "%format" strings within the album hierarchy, instead of just for track display.  I, like many, would like to have my albums displayed as (%year%) %album%.  At this point, I'm using the "solution" of writing (%year%) %album% in the comment tag, which isn't very useful.  However, I should just be able to re-write the album tagcache file (database_1.tcd) with those modifications.

I realize that mapping (%year%) %album% to the ALBUM tag is less flexible than actually allowing "%format" strings withing the database hierarchy (since any time the album tag is displayed in the database it would have to be displayed with the year).  As I understand, the database caches tags in the database_[0-8].tcd files.  I don't have much use for, say, the "composer" tag, or the "comment" tag.  What if we allowed users to define formats that are used in place of the actual tags?  The database cache files would only have to be written when the database is built, so the cost of this change is only incurred once.  For example, instead of storing the COMMENT tag in database_6.tcd, I could store something like this:

Code: [Select]

%format   "year_album"       "%d - %s" year album

Custom tag formats could go in a separate file (e.g. "tagnavi_setup.config") which is read only when the database is initialized or updated.  Then in my tagnavi file, I could use those formatted tags in place of actual tags, which I dont' use.

Granted, this suggestion is mostly syntactic sugar on top of the previous "solution" of just writing whatever I want to in, for instance, the COMMENT field.

Alternatively, there could just be more than 9 database cache files.  Perhaps files of the form database_user[0-9].tcd could define a set of these user-formatted tags.

Mike

[[Saint]]

Hi,


Editting the Database Index .tcd files in *NOT* the way to accomplish this.
Each time the database is rebuilt, your (probably quite painstaking to implement) changes will be lost.

The way to do this is via a custom Tagnavi file.

Tagnavi_Custom is loaded by RockBox and the values determined therein are accepted as though they were a part of the original tagnavi file.

A custom sorting order can be defined specifically for Artists/Albums beginning with "The", the rule can also include the command to strip X many characters from the display title (in your case 4 would be preferable) so that "The Beatles" would be displayed as "Beatles" etc.
Specific instructions on how to do this (and more) are available here:

http://www.rockbox.org/twiki/bin/view/Main/DataBase#tagnavi_config_v2_0_Syntax

(Practical working examples are nearer the bottom of the page, but be sure to read all of it as it's usefull information someones gone out of their way to compile)

If all that seems daunting to you, then another way of doing it would be to address the formatting of the files directly.

In the music library on my computer I have Albums and Artists beginning with "The" formatted so that the "The" is at the *end* of the title.

Example:

Instead of "The Beatles", in my library the folder (and more importantly for sorting in the RockBox Database the *Artist*) would be named "Beatles, The"

Most Media Players for the PC are intuitive enough to discount "The" from the title during sorting, but when viewing the actual folder/file structure of my music library I wasn't surprised to find that Windows Explorer *wasn't* intuitive enough to do so and "The Beatles" is included with the T's (and this is the problem you are having in RockBox from what I gathered, no?).

That slight annoyance prompted me to format the way I name Albums/Artists as "Beatles, The" and now not only is my music sorted correctly by Windows Explorer, as a result, they are also sorted correctly in the RockBox Database!!! 

If the idea of manually editing hundreds, thousands or possibly tens of thousands of .mp3's doesn't appeal to you, or seems daunting (for which I wouldn't blame you  ) there is a very simple and extremely versatile IDv3 Tag MetaData editor which (when it comes to my >500 Gigabyte Music Library) I find I almost certainly could not live without called Mp3Tag.

It is available for download here:

www.mp3tag.de/en/download.html

You'd probably be quite surprised (if you download a lot of music from varied sources) at the sheer amount of crap and missinformation that can be inserted into the Metadata of .mp3 files on a regular basis.

It can be very usefull to batch edit the metadata of your albums, as sometimes (more often than not I find) whichever Media Player you use (WMP, iTunes etc.) just plain gets the album info wrong and it can be a *real bastard* trying to correct the tag data afterwards.



I hope this information helps you to resolve this issue.



Sincerely,
[St.]

[wackymorningdj]

Hi [St.],

Thanks for the reply, but I think you misunderstood what I was going for.  I know that you can separate out "The" artists into a separate menu, and that I could re-tag all of my files so that "The Beatles" ends up as "Beatles, The", but (1) I want all of my artists in a single menu, not separated out by letter, and (2) I don't want "The Beatles" to show up in my wps as "Beatles, The".  I've read the tagnavi page forwards and backwards, and until you can use "%format" strings for custom formatting and sorting anywhere but in the final sub-menu (track display), a custom tagnavi file isn't going to cut it.  And probably the biggest reason that it's a pain is that I *do* have my music meticulously tagged.


So in the mean time, I wrote my own program to build the rockbox database, because I didn't feel like trying to wade my way through the rockbox tagcache source files.  It supports pretty much everything I originally wanted, including custom formats for any level of the database, sorting, and multiple value field handling.

Gui:       https://sourceforge.net/projects/rbdb/files/rbdbgui.zip/download
Source:  https://sourceforge.net/projects/rbdb/files/rbdb_src.zip/download


Mike

[[Saint]]

Wackymorningdj,


Sweet as man, sorry I missunderstood.

I'll download & check out your app (probably the best way for me to get an idea of it...) and see what it does.

I'm planning on trying to "clean up" as it were, the Database as I find it rather labyrinthine (personally) and a few of it's options/sub-menus surplus to my requirements.

Ultimately I want to try and "clean up" the whole menu structure (remove set & forget items, and things better handled by the application of a "Preferred Settings.cfg"), but that's going off topic 

Actually, it sounds as though you have a better understanding of the database structure than I do, so if you feel like giving me a hand with my simplified build you're *MORE* than welcome to...

WPS's I've got down to an art, but development/compiling (what I thought the next/natural progression to be) I am still *VERY* new to.


Thanks man,

Sincerely,
[St.]

[Chronon]

So in the mean time, I wrote my own program to build the rockbox database, because I didn't feel like trying to wade my way through the rockbox tagcache source files.  It supports pretty much everything I originally wanted, including custom formats for any level of the database, sorting, and multiple value field handling.

Gui:       https://sourceforge.net/projects/rbdb/files/rbdbgui.zip/download
Source:  https://sourceforge.net/projects/rbdb/files/rbdb_src.zip/download


Mike

I think this would make a nice addition to the UsefulTools page. 

[[Saint]]

A standalone version of the app would be awesome 

I mean, if you don't already have Python, & wxWidgets installed and wanted to use it...it'd kinda resemble bloatware, 'cos you'd be all like "Hmmm...Database Manager, awesome!...Oh, download this, this and this to use it."

Not saying it isn't cool, it is.

Or that I could do better, 'cos I can't.

But a standalone app with an installer & nice "rbutil-esque" GUI that did the same thing would be very, very cool.



[St.]

[wackymorningdj]

[St.]: I provided a compiled single-file (ok, two-files . . . there's a msvc dll in there) executable for windows already
https://sourceforge.net/projects/rbdb/files/rbdbgui.zip/download
I don't have a mac, so I can't make one for macs, though if anyone wanted to, they'd be more than welcome to, using py2app.  And I figure if you're comfortable using linux, it's probably not expecting much to install wxPython (I'm not sure there even is a py2elf compiler)

Chronon: sounds good to me.  I'll make a wiki page for it in the next few days.

Mike

[Chronon]

Thank you.