Dealing With Long Database Delays

[iPodVT]

I have some observations to share and questions to pose, regarding potential long delays when navigating Database->Track, and also when returning out of the WPS back to that Track list.  Please note that there is nothing in what follows that is intended to be criticism or complaint (I love Rockbox!).  I'm just hoping to contribute to improving Rockbox and also relate some things I've done on my end to work with things as they currently stand.

First the facts:

-iPod Video 5.5 Gen with 32MB RAM
-iFlash-Solo with Samsung Pro Plus 128GB microSD in an SD adapter
-3212 tracks in database - mostly 192 - 320 kbps AAC
-Today's dev build (230116)

When I navigate Database->Track it takes about 60 seconds to get to the list of all 3212 tracks.  When I click the Select button from there I get the "Loading (Play/Pause to abort)" progress bar and it takes about 10 seconds to get to the WPS and have the first track start to play.  If I click the Select button again while in the WPS it takes about another 60 seconds to return to the list of all 3212 tracks (so I assume it is having to regenerate the list on the fly).

Question: Why is there a progress bar for the 10 seconds delay but no progress bar for the 60 seconds delays during which the device gives the impression that it is frozen (though I have learned from experience that it isn't)?

Comment:  Despite the fact that the progress bar states "Play/Pause to abort", clicking that button does not result in an abort.

I have several times unintentionally/inadvertently entered that 60 seconds delay and cursed my carelessness.  So to try to help myself avoid making that mistake again I have modified my tagnavi_user.config file to insert an intermediate menu in between Database->Track and its action, which displays the title "Warning: This might take a long Time!" and contains a single menu item labeled "List All Tracks In Database".  This gives me a second chance to avoid the long delay.

Question: Once I am in the WPS, how can I resume navigating the Database browser without going through a 60 seconds return to the Tracks list?  If I click the Menu button to go back to the Main Menu, as soon I click Database I get sent into the 60 seconds return to the Tracks list.  My current means of getting around this is to shutdown and reboot the device.  (BTW - I LOVE the fact that it takes only a few seconds when I (re)boot into Rockbox, whereas with the same music library in the Apple firmware it takes a much, much longer time - and the same with disconnecting from my computer.)

I have made another related observation, with a non intuitive result, that I will describe in a subsequent post.

Thanks.

[Bilgus]

try Button Left not sure what that is on ipod

likely its missing the play/repeat

[iPodVT]

Yes - Button Left (not Play/Pause) does indeed trigger the abort.

Thanks!

[Bilgus]

also give you have plenty of ram have you tried using Load to Ram?
long press on the Database from main menu it should be the first option
I'm sorta surprised yo have such long delays with a sd card I wait about 15 seconds with 12000 tracks

[chris_s]

Was just about to say... 

I can mainly provide insight as a user of Rockbox. I've learned that it is essential to enable Load to RAM for the database (make sure to restart afterwards). This will make navigating around database menus instantaneous.

Using the database from disk is frustratingly slow, especially if a track is being buffered at the same time.

The only disadvantage of Load to Ram (besides obviously taking a bit of RAM) is that it will take some time for it to load after booting, during which the database can be navigated but will be even slower than normal. So you should wait for the disk activity indicator to disappear first.

Unfortunately, it can take *much* longer to load the database into RAM if the directory cache is enabled at the same time. This can be avoided by either disabling the directory cache (which I wouldn't necessarily recommend though) or by preventing dircache file references from bein loaded into the database. The latter requires patching Rockbox and is not currently an official option. It's how I've been using Rockbox for a long time. But I should run it by the more knowledgeable developers to consider whether this may be suited as an option for everybody, since it makes a huge difference, for me at least.

For reference, here are some stats on how long it takes for everything to load after the first button press to start the player on my iPod video using an iFlash Solo SD adapter with 20k audio tracks (and ~60k files in total)

- dircache/db_cache disabled: 7s
- dircache enabled: 18s
- db_cache enabled: 11s
- dircache+db_cache enabled: 1m 30s (!)
- dircache+db_cache enabled (without dircache refs): 22s

Dircache size: ~4MB
DB size: ~3MB

[Bilgus]

see when I disable dircache my clipzip runs like dog crap

I think better would be to implement incremental loading so you get into the menu and only rebuffer a screen or two at a time

but just storing the results (when possible) so when you return to the screen you don't have to wait again would be enough for me

[Bilgus]

dircache+db_cache enabled: 1m 30s (!)

i'd say this needs investigated they are probably fighting each other

[chris_s]

dircache+db_cache enabled: 1m 30s (!)

i'd say this needs investigated they are probably fighting each other

yeah, maybe. This is what causes the additional delay: https://github.com/Rockbox/rockbox/blob/aae34b2e7faead258e99e42bd199b329475eb17c/apps/tagcache.c#L4414

There's already a dircache_wait() before that though.

[iPodVT]

I did have Load to RAM turned on.  But I did not have Directory Cache turned on, so I turned it on, rebooted and timed the processes again:  Database->Track still took about a minute, as did returning to that Track list from the WPS.

[Bilgus]

yeah, maybe. This is what causes the additional delay: ...

There's already a dircache_wait() before that though.

yeah I was looking at that last night it looks to only be checking for deleted entries maybe we could move that out get the list up and check for validity once the user selects it if the scan isn't finished in a separate thread

@chris_s whats it do if you just disable the dircache part with #if 0
and let it use file_exists()?

[chris_s]

yeah, maybe. This is what causes the additional delay: ...

There's already a dircache_wait() before that though.

yeah I was looking at that last night it looks to only be checking for deleted entries maybe we could move that out get the list up and check for validity once the user selects it if the scan isn't finished in a separate thread

@chris_s whats it do if you just disable the dircache part with #if 0
and let it use file_exists()?

That's exactly what I'm doing with the last option mentioned above. Although my understanding is, that this means, the dcrefs aren't stored in the database at all and thus find_entry_ram has to always return -1. I've also disabled background scanning for the playlist.

In my personal experience, resulting negative effects are negligible. Tagcache_fill_tags won't work anymore but that seems like a good tradeoff, except, possibly, for the fact that a WPS skin that has an integrated playlist viewer can't display upcoming metadata anymore.

Would obviously be nicer to properly fix the slow loading times, if possible, and it seems misguided to add yet another option (e.g. Load to RAM: No / Yes / With Integrated Directory Cache). But I certainly prefer it on my own devices.

edit: or, actually, looking back at my code, I set rc_cache to 0, so I seem to have skipped the file_exists check completely in load_tagcache

[iPodVT]

In my personal experience, resulting negative effects are negligible. Tagcache_fill_tags won't work anymore but that seems like a good tradeoff, except, possibly, for the fact that a WPS skin that has an integrated playlist viewer can't display upcoming metadata anymore.

Would that affect the Next Track display that shows a track name and artist in my WPS?

[chris_s]

In my personal experience, resulting negative effects are negligible. Tagcache_fill_tags won't work anymore but that seems like a good tradeoff, except, possibly, for the fact that a WPS skin that has an integrated playlist viewer can't display upcoming metadata anymore.

Would that affect the Next Track display that shows a track name and artist in my WPS?

No. that would still work.

[iPodVT]

Cool.

[Bilgus]

this should be a bit faster as of today with https://gerrit.rockbox.org/r/c/rockbox/+/5075