Dealing With Long Database Delays

[Bilgus]

@iPodVT

I think I have a decent solution to very long delays with the database concerning the 'Tracks' selection
most of the time is spent formatting the list so I've added a submenu with sorted filenames, titles, and the slow title & duration (the existing 'Tracks' functionality)

https://gerrit.rockbox.org/r/c/rockbox/+/5118

I'll get it committed in the next few days after we've had chance to test it a bit

[iPodVT]

@iPodVT

I think I have a decent solution to very long delays with the database concerning the 'Tracks' selection
most of the time is spent formatting the list so I've added a submenu with sorted filenames, titles, and the slow title & duration (the existing 'Tracks' functionality)

https://gerrit.rockbox.org/r/c/rockbox/+/5118

I'll get it committed in the next few days after we've had chance to test it a bit

Thanks very much.  Is that a link to a test version I can try?  If so, I'm too thick to figure out where the download  link is.

[Bilgus]

No thats just a link to the patch
here is a iPodVideo Build if you want to try it
http://www.mediafire.com/file/acp38hf0qymwf20/iPodVideo_rockbox-full_TagNavi_tracks_g5118.zip

[iPodVT]

I tried it, and the Filename and Title options do indeed produce the list almost instantaneously with my ~3200 db entries, which is most excellent.

I was able to get similar results with my tagnavi_user.config file, as described in my post "Dealing With Long Database Delays - Part Two" [https://forums.rockbox.org/index.php/topic,54450.0.html].  My method also produced the list in about one second.

But my solution has the lousy shortcoming that it requires that the track files are all in a known location that is hard coded into my tagnavi_user.config file.  That was totally acceptable for my own personal use, but it's obviously not a good generalized solution.

Is there any difference between the list that is produced by the original method vs the list that is produced with the new 'Title' option?

[Bilgus]

the key here is skipping formatting but I see that you were still using formatting in yours
so there is clearly something still going on that might be worth investigating a bit further

[Bilgus]

interestingly doing "Title2" -> title ? filename ^ "/" -> title = "fmt_alphanum_title"
is indeed faster than
"Title mm:ss" -> title = "fmt_alphanum_title"

I'll try to figure out why before I commit this

[Edit*]
it appears adding the condition of filename makes the format no longer match so its doing the same thing
on the back end not totally sure why but the tagcache is a complicated opaque beast and it is quite hard to follow

you can observe in your track listing that there is nothing besides the track title
(and indeed in this patch the 'Title' item is the same)

whereas the item 'Title mm:ss' has the track duration and is about 10x slower

I'm guessing the reason formatting makes it slower is that you are
reading multiple tag files and its causing lots of random disk access
but I'm not totally sure if that is the whole story as I'd think the other list types would be affected too

[Bilgus]

after another few hours of playing with it..
the reason the other tag items are faster is because they don't do the
formatting over the whole tag corpus instead you drill it down to a specific artist or album
whereas 'Tracks' formats the whole shebang in a single operation

formatting is slow for several reasons:

• first it has to go through and find a matching formatter for the data it has that means string comparisons and or parsing
• second it has to parse the formatter it found to get the format items (%s %d etc)
• third it has to open/seek files to retrieve the tags in the format string

there is also a memset over a rather large buffer removing that shaves about 20 seconds off my results but i'm a bit wary to remove it

[Bilgus]

This is now merged:
https://gerrit.rockbox.org/r/c/rockbox/+/5118

the new options are about 10x faster

the formatted option is around 2x faster

[iPodVT]

With my 5.5Gen iPod (64MB RAM, iFlash-Solo with 256GB microSD in SD adapter), each of the three new "Tracks by" menu options now yields an almost instantaneous list.  Fantastic!  And this is with 12,025 entries in my complete database (as it currently stands).

For my part, it's unlikely that I will use the "Filename" option very much because all of my track filenames begin with track numbers, so an alphanumeric sort doesn't produce a particularly meaningful/useful order.  As well, since the "Title mm:ss" option seems to be as fast for me as the plain "Title" option, I see no advantage to using a list that doesn't include track time lengths.  But I am more than pleased to be able to have the choice of all of these now super speedy options.

Thank you so very much.