pictureflow problem - any suggestions for a fix/workaround?

[Frankenpod]

Thanks a lot, I'll give that version a try (still not in a position to compile my own patched version, linux PC still non-functional).

I don't know why it takes so long for me - even back when I had fewer tracks the picture-flow index-build took a very long time, to the point where I'd leave it on charge while building for fear the battery would otherwise run down before it finished.  I think I did get it to work eventually in those days though, but I think that was a 240gb HD mod and this is now flash modded so it's always possible that's the problem.

[Frankenpod]

Hurrah, your (tweaked) version works!

Downside is it still took 5 hours to finish the full index-building process!  I think previous times it fell over at the point of starting "preparing album art", which was at the 4 hour mark.  Don't know why it takes that long.  Some sort of memory/disk thrashing going on?

Seems as if each step takes about an hour (except for 'removing duplicates' which was a fairly quick one).

As I don't want to go through that again, what files should I copy off of the ipod to save/backup the picture-flow-specific database?

I'm assuming if/when I add any more tracks (and then update the database) the picture-flow plugin will stop working unless it's all rebuilt again?  Is there any way to 'update' the pictureflow database without recreating the entire thing?  Or can one continue to use the existing pictureflow even if the RB database has changed?

Also, is it at all feasable to make the size of the plug-in buffer a user-settable setting (via a menu option) rather than needing to compile a special version each time?  (Maybe with different defaults depending on the particular player's memory size).  Or is it something that unavoidably has to be hard-coded into the thing before compilation?

[chris_s]

That's good news. 

You didn't explicitly confirm or deny before, so I just want to make sure, since it is by far the most likely explanation for what you're experiencing:
Did you double-check that Load to RAM is enabled and that you've restarted the device afterwards, before attempting to build the index?

You can verify by going to System->Debug->View Database Info (RAM Cache should say "Yes") and by the fact that there should be zero perceptible lag when navigating through the database menus.

I've recently added some info to the PictureFlow manual entry to suggest as much, after I noticed that building the index is outrageously slow with the Load to RAM setting disabled (I'm a big user of the database, so I've always had it enabled). It makes a lot of sense to turn it on prior to building the index even if you should, later on, decide to turn it off again. It will actually speed up all aspects of using PictureFlow though, not jut the initial scan.

Once you can rule that out, it's time to start looking for another explanation.

Backing up the whole pictureflow folder as well as pictureflow.cfg stored in /.rockbox/rocks/demos/ gets you back to the current state at any time.

I don't feel particularly confident reasoning about the exact circumstances requiring a rebuild of the album index right now, maybe someone else can chime in on that, but rebuilding the index usually seems necessary after you've updated the Rockbox database. At the moment, the index is always re-created from scratch at that point.

It seems very unlikely to me that the plugin buffer size could be turned into a runtime setting, but I would again defer to someone else. As I wrote before, since the iPod classic (always?) has 64MB of RAM, it makes some sense to me give it a larger plugin buffer (i.e. 1 or 2 MiB – same as on the M3K) compared to the 32MB RAM iPods.

[Frankenpod]

This (successful) attempt I enabled "load database to RAM".  Not sure if that made a difference or not.  (My thought was that doing that would make any memory shortfall problem worse rather than better, as it would be using some of the RAM for the database itself...but I don't really have much idea what is going on so that is probably mistaken).

However, now I power it up again and check the debug menu, RAM Cache says 'no'!  Puzzled as I enabled it in the menu option and then rebooted.

Apart from that 'removing duplicates' step, the building process was so slow you couldn't tell the progress bar was progressing, would have to go away for half-an-hour and come back to be able to tell it hadn't just frozen.

Would there be a downside to having some sort of Macro(?) or IFDEF(?) sort of statement in the code (I have only a vague idea about such things) to always use a larger plugin-buffer for targets with more RAM?  Might not be justifiable to do that as it seems to be a problem that only comes up with mods with more than the original HD capacity.  Seems like the issue is just the sheer number of tracks being indexed.  Though to be sure that it's the number of tracks and not an iFlash disk-writing issue, I'd have to experiment with building pictureflow index on an iFlash modded ipod with only a small number of tracks on it.

Is there any way to show the artist name in the pictureflow display, as well as the album title?

[Frankenpod]

Thanks for building the patched version for me, btw.

[chris_s]

Happy to do it.

Loading the database into RAM takes space away from the audio buffer, but not from the plugin buffer, so it should only offer advantages from PictureFlow's perspective.

As for why it says "No" in the Debug menu- maybe you haven't waited long enough after restarting before checking the menu? Considering your database size,  it may take a few minutes if the disk cache is enabled as well. Having Both options activated together takes much longer than each option individually.

The plugin buffer size is already easily changed per-model. Literally all I had to do was change L140 to #define PLUGIN_BUFFER_SIZE  0x200000 . Could probably be made dependent on the RAM size as well.

You can set both album and artist to be displayed under "Show album title" in the settings menu for PictureFlow.

[Frankenpod]

I don't see an option for showing the artist name...

[fiddles with the player a bit more]

Ah it's a sub-option under 'show album title'.

Nice that it now allows sorting by artist-then-album.  Long wanted that option.

Something I was thinking was that maybe it would be possible to 'morph' the pictureflow plugin to one that allows you to navigate through album cover thumbnails laid out in a grid format, like the Sony players firmware does?  It would just be a different way of displaying the covers, could presumably use the same database/index?

Will try building the pictureflow DB again on another player, this time trying to make sure the 'load DB to RAM' option has fully 'taken' before I do it.  I find that various options sometimes don't "take" for reasons I can't figure out (e.g. increasing the maximum number of entries in file view has that problem).  I think maybe it needs the player to be actually shut down and restarted rather than just rebooted for some options to actually kick in.

[Frankenpod]

Meh, I don't know what's going on with it.  I can set "load database to RAM" but the debug/databaseinfo/RAM cache remains set to "No", no matter how many times I reboot or restart.

[chris_s]

Have you tried turning the dircache off to see whether that changes anything?

Here's a build with logging enabled for the tagcache: If you want to try that, it could hopefully help us narrow down where it's running into problems (not that I would be able to fix the issue in all likelihod, lol, but someone might...) .

You should see additional options to "Show Log File" and "Dump Log File" in the Debug menu after installing this build. It's probably easiest if you post the dumped log file after enabling Load to RAM and restarting.

https://www.dropbox.com/s/b8iau1snqiwi9kp/rockbox-full-ipodclassic-logf.zip?dl=0

[Frankenpod]

So the plan is to install that build on the ipod, then build RB database and check the log?  Or do you mean to build the pictureflow database and _then_ check the log?

Will give it a go, either way.

Thanks for your efforts in looking into this (even if you won't be able to fix whatever the heck the problem is).


Is the debug/databaseinfo "RAM cache" the same thing that "load database to RAM" is supposed to be setting?  Because it's odd, the "load database to RAM" accepts the setting to 'on' and _stays_ set to 'on', but it doesn't seem to change what it says in the debug menu.

[chris_s]

The plan would be that after you've installed the build, have built the RB database and have enabled "Load to RAM", you then restart and look at (or dump) the log after Rockbox had some time to process. Hopefully that will tell us something useful. A normal output would look something like this:

Code: [Select]

Tagcache: 3935008 bytes allocated.
Loading tagcache to ram...
Tagcache loaded into ram!
Utilization 99%
Tagcache check done
You don't have to launch PictureFlow at all for that (let alone go through rebuilding the index again) , since my suggestion would be that you first try to get Load to RAM working for the database.

Yes, "load database to RAM" and "RAM cache" refers to the same thing there. The fact that it never changes to YES just shows that Rockbox seems to be having trouble loading the database into RAM for one reason or another.

[Frankenpod]

OK, got it.  Will post the log file when done.  Thanks again.

[Frankenpod]

OK, it seems to have solved itself, but I don't entirely understand how.

  Trying to build the database and see what the log said (with multiple different flash-modded ipods) proved more difficult and time-consuming than I expected.  Firstly because doing a full database build takes quite a long time, but secondly because it kept producing weird results - the log file often claimed to have been unable to open the necessary database files, even though the database ended up being built.  Even when it didn't do that it never said anything about loading to RAM even though that option was selected.  A couple of times the database build process would hang at the first 'commit' step.  Or claim to have finished and then I'd find there was no database there and would have to do it again.  I think this may just be the usual glitches with flash mods.

(Plus I'm not clear whether it's necessary to set 'load to RAM' before building the database, or if when you select that the idea is it will load an already existing database from disk to RAM.)

However, after trying several times, the 'load to RAM' thing suddenly worked, and the debug menu suddenly reported the cache as existing.  I _think_ it was a question of leaving it for a time after building (or repeatedly scanning through the database entries?) until it decided to actually cache it.

Or it might have been just that particular ipod (maybe the brand of memory card in it?).  Need to try to get the same result with other ipods.

Once it actually loaded the database to RAM I again tried building the picture flow index, and this time it completed the whole process in 35 minutes (most of it being the "preparing artwork" step) still a fairly long time, but a massive improvement over the 5 hours it took without the database in RAM.

[Frankenpod]

So the one favour I'd hope for now is that the tweak to increase plug-in buffer size for ipod classics (and video?) could be 'mainlined' into the dev versions so I won't have to specifically patch them for each new dev or release version from this point on.  I hope there's no downside to doing this?

(As I understand it, for the 5th and 5.5 gen ipods, the ones that came originally with 30Gb drives had 32Mb RAM, and the ones with 60Gb or more had 64Mb, as did all the 6th,7th and 7.5 gens...so maybe there's a case for being more conservative with the 30GB models?)

[chris_s]

Nice. 

OK, it seems to have solved itself, but I don't entirely understand how.

  Trying to build the database and see what the log said (with multiple different flash-modded ipods) proved more difficult and time-consuming than I expected.  Firstly because doing a full database build takes quite a long time, but secondly because it kept producing weird results - the log file often claimed to have been unable to open the necessary database files, even though the database ended up being built.  Even when it didn't do that it never said anything about loading to RAM even though that option was selected.  A couple of times the database build process would hang at the first 'commit' step.  Or claim to have finished and then I'd find there was no database there and would have to do it again.  I think this may just be the usual glitches with flash mods.

Huh. It may also sometimes be necessary to restart the device after building the database for the changes to be committed.

(Plus I'm not clear whether it's necessary to set 'load to RAM' before building the database, or if when you select that the idea is it will load an already existing database from disk to RAM.)

Sorry if I've been unclear about that. It's the latter. You can change that setting back and forth at any time, even if the database has already been built. It only applies after a reboot though (and then requires waiting for the loading process to finish before the database's "RAM Cache" entry in the debug menu will change to "Yes" if Load to RAM has been enabled).

Once it actually loaded the database to RAM I again tried building the picture flow index, and this time it completed the whole process in 35 minutes (most of it being the "preparing artwork" step) still a fairly long time, but a massive improvement over the 5 hours it took without the database in RAM.

This seems in line with what I would expect for a (very) large library with a lot of artwork, if you include the preparing artwork step.