Rockbox Technical Forums

Support and General Use => Plugins/Viewers => Topic started by: Frankenpod on October 25, 2021, 04:35:46 AM

Title: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on October 25, 2021, 04:35:46 AM
I can't build the database for pictureflow, because it goes for ages then fails with an 'insufficient memory' error.  The error is more specific than that, it refers to a particular kind of memory, but without spending ages letting it run again I can't remember what the exact error message was, sorry.

I presume it's because this is a flash modded ipod and has a lot of tracks (> 10,000) so a lot of cover art.

  Alternatively, possibly it's to do with the usual problem with writing to the drive on flash-modded ipods...but my impression is that has gotten a lot better with later dev versions (this was using a very recent dev version, i.e. within the last couple of weeks...it seems more recent versions are more stable than RB used to be when it comes to building the database on modded ipods, for example, so I'm wondering if now the problem with pictureflow is simply what it says it is - too much cover art for the allocated memory?).

Does anyone have any ideas as to how to work around this?

I might try a test with an iFlash ipod with not many tracks on it, to check if it really is just a matter of too much cover art for the plug-in to cope with.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Bilgus on November 04, 2021, 12:24:12 PM
Pictureflow uses the remaining plugin buffer to create the list of tracks and then writes it to a database file
I recently rewrote it to do t :o :(his in in stages likely it's just out of space..
We could probably stop playback and take the audio buffer too if it's not already doing that
During building

I don't remember if the album art is resized before pf copies it but if so that could be an issue
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on November 22, 2021, 01:17:33 PM
I notice there have been some fixes to pictureflow by Chris_s recently.  Would those relate to the kind of crash I was getting?  i.e. is it worth trying again with a recent dev version of rockbox?  Or were those fixing some other, unrelated, bug/crash issue?

On a related note, as there seems to have been a lot of changes (some reverted?) to both the database handling and pictureflow in recent dev versions, I don't suppose there's any chance anyone would do an update of the Windows-application simulator builds (as downloadable under 'extras')?

 Perhaps that's asking a lot as I guess it would be a lot of work to build those for every player.  But the changes to the database code means that one can't now use the simulators (as currently available) to build the database on the PC and copy it to the device.

(I notice at least at one point, the database lost the separate artist/album artist tag choices...curious what the reasoning for that was.  It's not a big deal, though usually I'd prefer to use the 'album artist' tag, as it avoided including entries for large numbers of ultra-obscure artists that I just happened to have one lone track from on a compilation album - making it easier to find the artist I was actually looking for...and conversely, sometimes I'd use the 'artist' field, if I was looking for one of those obscure compilation-album-only artist tracks.)

Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: chris_s on November 23, 2021, 01:57:15 AM
I don't think my PF fixes will be helpful with respect to the crashes you were getting building the album index. Have you tried using smaller artwork files as Bilgus, I think, alluded to? I have about 17k songs and neither the iPod 4g nor iPod video (both modded using an "iFlash-Solo" adapter)  seem to have trouble building the index on device (although it takes a long time and I have to make sure the device doesn't go to sleep in between) with 160x160 JPEGs.

As for the album artist entry, I'm with you, it's also been discussed on IRC and Gerrit. Like you said, PictureFlow at least has been reverted to using album artists again. Which may mean, although I could very well be wrong, that the old Windows Simulator applications can once again be used to build a PF database for the latest dev builds... I don't know who'd be able to update them otherwise..
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on November 24, 2021, 01:35:02 AM
Thanks (and thanks for your ongoing work on Rockbox).

I guess I could try shrinking the artwork files - would have to set up some series of scripts to extract all the embedded art, then resize it, then re-embed it, much as I did when purging all the .png format art, but it would be a bit of effort and haven't got round to trying it yet.  Presently most of the art is about 300x300 pixels, not sure how much shrinking would be needed to get picture flow DB to build - and I'm not certain a memory-shortage issue is the only problem there anyway.

  When it gives an error message (right at the very end after taking an absolute age to go through the stages of building the pictureflow DB) about 'insufficient memory', is it referring to onboard RAM, or does it mean it's having trouble writing stuff to the flash card?
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: chris_s on November 24, 2021, 02:23:38 AM
For what it’s worth, the PF plugin, unlike the playback engine of Rockbox, doesn’t look at embedded artwork, only at separately stored artwork files, so you should probably make sure the latter are of an appropriate size. The order used by Rockbox is also described in the manual: https://download.rockbox.org/daily/manual/rockbox-ipod6g/rockbox-buildap3.html#x19-430000C

My understanding would be that “insufficient memory” refers to the plugin buffer only (so part of RAM), but Bilgus is obviously better equipped to speak on that.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on November 25, 2021, 05:07:17 AM
For what it’s worth, the PF plugin, unlike the playback engine of Rockbox, doesn’t look at embedded artwork, only at separately stored artwork files, so you should probably make sure the latter are of an appropriate size.

That's helpful to know.  At some point I guess I'll try bulk-resizing all the cover.jpg files (leaving the embedded art untouched - I mostly have cover.jpg files in each directory with just the first track having the same art embedded in it), and see if that makes any difference with building the PF DB.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on November 27, 2021, 07:57:12 AM
Drat.  Even after downsizing every cover.jpg to 160 pixels width, I still get the 'insufficient memory for album art cache' error message at the end of the initialising process.  Not sure what is the issue.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: amachronic on November 27, 2021, 09:14:42 AM
"Not enough memory for album art cache" means the artist & album index is too big and there is not enough memory left over for buffering the cover images as they are loaded. You can look for the pictureflow_album.idx file (it'll be somewhere under the .rockbox folder) and see how big it is. The size of that file should be approximately how much RAM is being used for the index. If it's too big, that'd be the problem.

The size of your cover art shouldn't matter too much -- I think it's only read in small chunks and gets re-scaled on the fly.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on November 28, 2021, 05:21:14 PM
Thanks.  I'll have a look and see if it's there and what size it is - assuming it actually succeeded in creating that file - it seemed as if the whole initialisation process failed at that point, but not really sure.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: chris_s on December 13, 2021, 06:12:11 PM
I've noticed the plugin buffer size is only set to 800KiB on all iPods including the iPod classic, despite it having 64MB of RAM, whereas, for example, amachronic seems to have set it to 2 MiB on the M3K with the same amount of RAM. Might be worth trying to simply increase that at bit, at least, in a one-off build for yourself?
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: chris_s on December 13, 2021, 06:18:16 PM
Or 512KiB actually...
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on December 13, 2021, 10:48:16 PM
Will try that - though, unfortunately, my linux PC (that I'd have to use to build RB with) is currently partially-dismantled!
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 08, 2022, 12:33:19 PM
Gave it another go with the latest RB version.  Took 4 hours to complete then said 'not enough memory for art cache'!

the pictureflow_album.idx file is 304k, which doesn't seem that large.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: chris_s on May 08, 2022, 04:19:20 PM
Gave it another go with the latest RB version.  Took 4 hours to complete then said 'not enough memory for art cache'!

the pictureflow_album.idx file is 304k, which doesn't seem that large.
Doesn't really solve your problem, but building the index shouldn't take that long. Have you enabled 'Load to RAM for the database (and restarted afterwards)? I have just under 20k songs on my iPod 4G, resulting in an album index that's about 130 KB, and it takes  (I just timed it) under 2 minutes to build the index and reach the "Preparing artwork" stage.

You're using an iPod 6G, if I remember correctly? Here's a dev build with the plugin buffer size set to 2MiB instead of 512KiB, if you'd like to give it a go: https://www.dropbox.com/s/tyi960dv0halide/rockbox-full-ipodclassic.zip?dl=0
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 08, 2022, 05:49:05 PM
Thanks a lot, I'll give that version a try (still not in a position to compile my own patched version, linux PC still non-functional).

I don't know why it takes so long for me - even back when I had fewer tracks the picture-flow index-build took a very long time, to the point where I'd leave it on charge while building for fear the battery would otherwise run down before it finished.  I think I did get it to work eventually in those days though, but I think that was a 240gb HD mod and this is now flash modded so it's always possible that's the problem.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 08, 2022, 11:48:47 PM
Hurrah, your (tweaked) version works!

Downside is it still took 5 hours to finish the full index-building process!  I think previous times it fell over at the point of starting "preparing album art", which was at the 4 hour mark.  Don't know why it takes that long.  Some sort of memory/disk thrashing going on?

Seems as if each step takes about an hour (except for 'removing duplicates' which was a fairly quick one).

As I don't want to go through that again, what files should I copy off of the ipod to save/backup the picture-flow-specific database?

I'm assuming if/when I add any more tracks (and then update the database) the picture-flow plugin will stop working unless it's all rebuilt again?  Is there any way to 'update' the pictureflow database without recreating the entire thing?  Or can one continue to use the existing pictureflow even if the RB database has changed?

Also, is it at all feasable to make the size of the plug-in buffer a user-settable setting (via a menu option) rather than needing to compile a special version each time?  (Maybe with different defaults depending on the particular player's memory size).  Or is it something that unavoidably has to be hard-coded into the thing before compilation?
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: chris_s on May 09, 2022, 04:24:06 AM
That's good news.  :)

You didn't explicitly confirm or deny before, so I just want to make sure, since it is by far the most likely explanation for what you're experiencing:
Did you double-check that Load to RAM (https://download.rockbox.org/daily/manual/rockbox-ipod6g/rockbox-buildch4.html#x7-480004.2.3) is enabled and that you've restarted the device afterwards, before attempting to build the index?

You can verify by going to System->Debug->View Database Info (RAM Cache should say "Yes") and by the fact that there should be zero perceptible lag when navigating through the database menus.

I've recently added some info to the PictureFlow manual entry (https://download.rockbox.org/daily/manual/rockbox-ipod6g/rockbox-buildch12.html#x15-26900012.2.12) to suggest as much, after I noticed that building the index is outrageously slow with the Load to RAM setting disabled (I'm a big user of the database, so I've always had it enabled). It makes a lot of sense to turn it on prior to building the index even if you should, later on, decide to turn it off again. It will actually speed up all aspects of using PictureFlow though, not jut the initial scan.

Once you can rule that out, it's time to start looking for another explanation.

Backing up the whole pictureflow folder as well as pictureflow.cfg stored in /.rockbox/rocks/demos/ gets you back to the current state at any time.

I don't feel particularly confident reasoning about the exact circumstances requiring a rebuild of the album index right now, maybe someone else can chime in on that, but rebuilding the index usually seems necessary after you've updated the Rockbox database. At the moment, the index is always re-created from scratch at that point.

It seems very unlikely to me that the plugin buffer size could be turned into a runtime setting, but I would again defer to someone else. As I wrote before, since the iPod classic (always?) has 64MB of RAM, it makes some sense to me give it a larger plugin buffer (i.e. 1 or 2 MiB – same as on the M3K) compared to the 32MB RAM iPods.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 09, 2022, 07:03:02 AM
This (successful) attempt I enabled "load database to RAM".  Not sure if that made a difference or not.  (My thought was that doing that would make any memory shortfall problem worse rather than better, as it would be using some of the RAM for the database itself...but I don't really have much idea what is going on so that is probably mistaken).

However, now I power it up again and check the debug menu, RAM Cache says 'no'!  Puzzled as I enabled it in the menu option and then rebooted.

Apart from that 'removing duplicates' step, the building process was so slow you couldn't tell the progress bar was progressing, would have to go away for half-an-hour and come back to be able to tell it hadn't just frozen.

Would there be a downside to having some sort of Macro(?) or IFDEF(?) sort of statement in the code (I have only a vague idea about such things) to always use a larger plugin-buffer for targets with more RAM?  Might not be justifiable to do that as it seems to be a problem that only comes up with mods with more than the original HD capacity.  Seems like the issue is just the sheer number of tracks being indexed.  Though to be sure that it's the number of tracks and not an iFlash disk-writing issue, I'd have to experiment with building pictureflow index on an iFlash modded ipod with only a small number of tracks on it.

Is there any way to show the artist name in the pictureflow display, as well as the album title?
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 09, 2022, 07:03:41 AM
Thanks for building the patched version for me, btw.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: chris_s on May 09, 2022, 07:29:19 AM
Happy to do it.

Loading the database into RAM takes space away from the audio buffer, but not from the plugin buffer, so it should only offer advantages from PictureFlow's perspective.

As for why it says "No" in the Debug menu- maybe you haven't waited long enough after restarting before checking the menu? Considering your database size,  it may take a few minutes if the disk cache is enabled as well. Having Both options activated together takes much longer than each option individually.

The plugin buffer size is already easily changed per-model. Literally all I had to do was change L140 (https://github.com/Rockbox/rockbox/blob/4b293285ea04df6c8ca04ec01f1b4729d285d379/firmware/export/config/ipod6g.h#L140) to #define PLUGIN_BUFFER_SIZE  0x200000 . Could probably be made dependent on the RAM size as well.

You can set both album and artist to be displayed under "Show album title" in the settings menu for PictureFlow.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 09, 2022, 08:17:12 AM
I don't see an option for showing the artist name...

[fiddles with the player a bit more]

Ah it's a sub-option under 'show album title'.

Nice that it now allows sorting by artist-then-album.  Long wanted that option.

Something I was thinking was that maybe it would be possible to 'morph' the pictureflow plugin to one that allows you to navigate through album cover thumbnails laid out in a grid format, like the Sony players firmware does?  It would just be a different way of displaying the covers, could presumably use the same database/index?

Will try building the pictureflow DB again on another player, this time trying to make sure the 'load DB to RAM' option has fully 'taken' before I do it.  I find that various options sometimes don't "take" for reasons I can't figure out (e.g. increasing the maximum number of entries in file view has that problem).  I think maybe it needs the player to be actually shut down and restarted rather than just rebooted for some options to actually kick in.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 09, 2022, 11:10:12 AM
Meh, I don't know what's going on with it.  I can set "load database to RAM" but the debug/databaseinfo/RAM cache remains set to "No", no matter how many times I reboot or restart.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: chris_s on May 09, 2022, 12:32:35 PM
Have you tried turning the dircache off to see whether that changes anything?

Here's a build with logging enabled for the tagcache: If you want to try that, it could hopefully help us narrow down where it's running into problems (not that I would be able to fix the issue in all likelihod, lol, but someone might...) .

You should see additional options to "Show Log File" and "Dump Log File" in the Debug menu after installing this build. It's probably easiest if you post the dumped log file after enabling Load to RAM and restarting.

https://www.dropbox.com/s/b8iau1snqiwi9kp/rockbox-full-ipodclassic-logf.zip?dl=0
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 09, 2022, 03:00:24 PM
So the plan is to install that build on the ipod, then build RB database and check the log?  Or do you mean to build the pictureflow database and _then_ check the log?

Will give it a go, either way.

Thanks for your efforts in looking into this (even if you won't be able to fix whatever the heck the problem is).


Is the debug/databaseinfo "RAM cache" the same thing that "load database to RAM" is supposed to be setting?  Because it's odd, the "load database to RAM" accepts the setting to 'on' and _stays_ set to 'on', but it doesn't seem to change what it says in the debug menu.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: chris_s on May 09, 2022, 03:17:28 PM
The plan would be that after you've installed the build, have built the RB database and have enabled "Load to RAM", you then restart and look at (or dump) the log after Rockbox had some time to process. Hopefully that will tell us something useful. A normal output would look something like this:

Code: [Select]
Tagcache: 3935008 bytes allocated.
Loading tagcache to ram...
Tagcache loaded into ram!
Utilization 99%
Tagcache check done

You don't have to launch PictureFlow at all for that (let alone go through rebuilding the index again) , since my suggestion would be that you first try to get Load to RAM working for the database.

Yes, "load database to RAM" and "RAM cache" refers to the same thing there. The fact that it never changes to YES just shows that Rockbox seems to be having trouble loading the database into RAM for one reason or another.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 09, 2022, 05:46:39 PM
OK, got it.  Will post the log file when done.  Thanks again.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 09, 2022, 09:13:06 PM
OK, it seems to have solved itself, but I don't entirely understand how.

  Trying to build the database and see what the log said (with multiple different flash-modded ipods) proved more difficult and time-consuming than I expected.  Firstly because doing a full database build takes quite a long time, but secondly because it kept producing weird results - the log file often claimed to have been unable to open the necessary database files, even though the database ended up being built.  Even when it didn't do that it never said anything about loading to RAM even though that option was selected.  A couple of times the database build process would hang at the first 'commit' step.  Or claim to have finished and then I'd find there was no database there and would have to do it again.  I think this may just be the usual glitches with flash mods.

(Plus I'm not clear whether it's necessary to set 'load to RAM' before building the database, or if when you select that the idea is it will load an already existing database from disk to RAM.)

However, after trying several times, the 'load to RAM' thing suddenly worked, and the debug menu suddenly reported the cache as existing.  I _think_ it was a question of leaving it for a time after building (or repeatedly scanning through the database entries?) until it decided to actually cache it.

Or it might have been just that particular ipod (maybe the brand of memory card in it?).  Need to try to get the same result with other ipods.

Once it actually loaded the database to RAM I again tried building the picture flow index, and this time it completed the whole process in 35 minutes (most of it being the "preparing artwork" step) still a fairly long time, but a massive improvement over the 5 hours it took without the database in RAM.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 09, 2022, 09:28:06 PM
So the one favour I'd hope for now is that the tweak to increase plug-in buffer size for ipod classics (and video?) could be 'mainlined' into the dev versions so I won't have to specifically patch them for each new dev or release version from this point on.  I hope there's no downside to doing this?

(As I understand it, for the 5th and 5.5 gen ipods, the ones that came originally with 30Gb drives had 32Mb RAM, and the ones with 60Gb or more had 64Mb, as did all the 6th,7th and 7.5 gens...so maybe there's a case for being more conservative with the 30GB models?)
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: chris_s on May 10, 2022, 01:53:47 AM
Nice.  :)
OK, it seems to have solved itself, but I don't entirely understand how.

  Trying to build the database and see what the log said (with multiple different flash-modded ipods) proved more difficult and time-consuming than I expected.  Firstly because doing a full database build takes quite a long time, but secondly because it kept producing weird results - the log file often claimed to have been unable to open the necessary database files, even though the database ended up being built.  Even when it didn't do that it never said anything about loading to RAM even though that option was selected.  A couple of times the database build process would hang at the first 'commit' step.  Or claim to have finished and then I'd find there was no database there and would have to do it again.  I think this may just be the usual glitches with flash mods.
Huh. It may also sometimes be necessary to restart the device after building the database for the changes to be committed.

(Plus I'm not clear whether it's necessary to set 'load to RAM' before building the database, or if when you select that the idea is it will load an already existing database from disk to RAM.)

Sorry if I've been unclear about that. It's the latter. You can change that setting back and forth at any time, even if the database has already been built. It only applies after a reboot though (and then requires waiting for the loading process to finish before the database's "RAM Cache" entry in the debug menu will change to "Yes" if Load to RAM has been enabled).


Once it actually loaded the database to RAM I again tried building the picture flow index, and this time it completed the whole process in 35 minutes (most of it being the "preparing artwork" step) still a fairly long time, but a massive improvement over the 5 hours it took without the database in RAM.
This seems in line with what I would expect for a (very) large library with a lot of artwork, if you include the preparing artwork step.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 10, 2022, 06:42:33 AM
Yeah, I am convinced part of the problem is the usual glitches with flash-mods and writing to disk.  It seems to work better with the micro-SD-card/Quad adaptor ones.  The ones that use full size cards I find I'll go through the whole database build, the database will be there and accessible, then I'll set 'load to RAM', then when I go back to the database again it's disappeared and insists on rebuilding itself entirely!  That sort of thing was why I used to build the database with the simulator on the PC.  Problem now is I'm not sure the simulators for download use an up-to-date version of RB.

Had more luck with the quad-adaptor ipods (managed to get pictureflow working perfectly well on two of them now), though still haven't worked out what it is that causes the database caching to actually kick in.  It seems rather random.  Definitely seems it makes a big difference to how long the cover art indexing takes, so I'll just have to keep checking the debug menu to see when the cache has actually started doing its thing.

In terms of what the underlying code actually says, when you select 'load database to RAM', at what point does it in fact_do_ that?   Is it supposed to do it immediately you select the option, or is it deferred in some way, till something triggers it to happen?
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 10, 2022, 06:58:33 AM

 It may also sometimes be necessary to restart the device after building the database for the changes to be committed.


For whatever reason, neither rebooting (i.e. holding centre button+menu) nor restarting (holding play till it shuts down then starting up again) seems to reliably cause that 'database cache' to activate.  I don't really know _how_ I got it to work (twice).  It's proving quite mysterious.  (Perhaps it's a hardware issue I'm having?).  But certainly it makes a difference to the time it takes to build the art database.   In addition, the increase you made to the plug-in buffer size seems to be essential to be able to build it at all.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: chris_s on May 10, 2022, 08:27:22 AM
Yeah, I am convinced part of the problem is the usual glitches with flash-mods and writing to disk.

It's possible, although, at least in my experience with an iPod video and iPod 4G, each with an iFlash Solo and 512GB SD card, they've been rock-solid ever since speachy et al improved the compatibility of the power management handling – ymmv.

In terms of what the underlying code actually says, when you select 'load database to RAM', at what point does it in fact_do_ that?   Is it supposed to do it immediately you select the option, or is it deferred in some way, till something triggers it to happen?

Changing the option doesn't seem to trigger anything (e.g. if the database was in RAM before, it will still be loaded after you've turned the setting off, at least until you update the database or reboot). From quick peeks at the code, I would say it is required to reboot to make sure sufficient space for the database RAM cache has been allocated (https://github.com/Rockbox/rockbox/blob/4b293285ea04df6c8ca04ec01f1b4729d285d379/apps/tagcache.c#L4868) and can be utilized.



 It may also sometimes be necessary to restart the device after building the database for the changes to be committed.


For whatever reason, neither rebooting (i.e. holding centre button+menu) nor restarting (holding play till it shuts down then starting up again) seems to reliably cause that 'database cache' to activate. 

Yeah, that part seems weird. Just to mention it though, and sorry if this is very obvious, but you will have to wait the same amount of time for the database to be loaded into RAM each time your player boots, even if the setting hasn't changed. As I mentioned before, the way the disk cache and database cache interact, means it can take a while for the combination of the two to load – certainly much longer than it takes for the two to load separately. I usually wait for the HD indicator (if your theme displays one) to disappear to know it's now "safe" to use the DB.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 10, 2022, 08:40:59 AM
Well, I'm getting there - now have four (quad) flash ipods with pictureflow working.  Trouble is I might lose the pictureflow capacity next time I resync them and have to do it again.  Still can't work out what it is that actually causes the database to be loaded to RAM. 

It seems that maybe it happens if you browse the database and scroll through the whole thing...not sure.

Also, it would be handy if one could set the "context menu plug-in" using the fixed.cfg file, so it would be pictureflow by default.  Some things, like battery capacity and file-browser max items, I found helpful to copy from config.cfg to fixed.cfg, but the selected context menu plug-in doesn't seem to get saved in config.cfg so I can't copy the entry into fixed.cfg, and hence have to reset the plug-in to pictureflow each reboot.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: chris_s on May 10, 2022, 08:51:24 AM
In theory you shouldn't have to do anything but wait (after rebooting) for the cache to load – no idea what's going on there.  :D

Re fixed.cfg – I'm only barely aware of that functionality. Seems useful, but shouldn't it still remember the context plugin after a reboot, even if some other setting get reset to the values specified in fixed.cfg?
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 10, 2022, 08:59:32 AM
In theory you shouldn't have to do anything but wait (after rebooting) for the cache to load – no idea what's going on there.  :D

Re fixed.cfg – I'm only barely aware of that functionality. Seems useful, but shouldn't it still remember the context plugin after a reboot, even if some other setting get reset to the values specified in fixed.cfg?

Yeah, for whatever reason it doesn't seem to remember the context menu plugin.  Other settings get saved in config.cfg or can, more permanently, be stored in fixed.cfg.  I resorted to using fixed.cfg to keep the max file browser entries at 9000, because it otherwise seemed to keep reseting it to 1000 (nowhere near large enough!).  Don't know if that's expected RB behaviour or if it's another iflash glitch.

Can't find information on where the context menu plug-in is recorded - would have expected it to be in config.cfg, but it doesn't seem to be.

Edit - ah, seems it's in rockbox/rocks/plugin.dat.  Or at least it is on a HD-based ipod.  Perhaps it's yet another issue with writing to flash drives?  Nope, it's there on the iflash ipods also, but for some reason RB doesn't seem to be paying attention to it unless I reset it again.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 10, 2022, 10:18:19 AM
Weird - now it's started remembering the context plug-in, no longer insisting I set it again every reboot.  I give up trying to understand what's going on!
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Bilgus on May 10, 2022, 01:44:57 PM
I still have some work to do with the openplugins module its independent of the cfg file but is keyed to langids so when the lang file changes it breaks the core ones its nothing major or dangerous just annoying but I haven't decided on a better way yet probably just an export function and better versioning
Id hash language stings but it breaks when switching languages then.

Setting the context plugin is immediate but it won't get saved permanantely unless you do a safe shutdown or leave it on long enough that the device decides to flush to disk and it saves to a temp file first like the cfg files do on flash devices
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 10, 2022, 06:07:07 PM
I still have some work to do with the openplugins module its independent of the cfg file but is keyed to langids so when the lang file changes it breaks the core ones its nothing major or dangerous just annoying but I haven't decided on a better way yet probably just an export function and better versioning
Id hash language stings but it breaks when switching languages then.

Setting the context plugin is immediate but it won't get saved permanantely unless you do a safe shutdown or leave it on long enough that the device decides to flush to disk and it saves to a temp file first like the cfg files do on flash devices

To be honest, don't think I entirely understand all that, but it sounds as if that explains why sometimes the device remembers what plug-in I previously set it to use (picture flow) and other times it seems to forget so I have to set it again.

I think ideally the context menu would actually say the name of the currently selected context-menu plug-in rather than 'open plug in', incidentally.

[Edit] That's not a complaint or anything.  I'm just a bit less baffled now.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 11, 2022, 03:28:13 PM
Yes! Finally managed to persuade the last iflash ipod to enable database cache.  It definitely makes a huge difference to the speed of building pictureflow index.

But I still haven't manage to work out what it is that causes database caching to suddenly become active.  Have yet to find a pattern to it.

  Seems to be the procedure is: 'turn on load database to RAM, try browsing the database repeatedly, reboot a few times, check debug menu to find it's still not caching the database, browse the database from start to end in hope of persauding it to start using the cache, to no avail, try building pictureflow index and abort it again because it's so painfully slow, give up and leave it alone, come back to it, check to find it's still not caching the DB, reboot it a couple more times...and discover its now finally decided to cache the database, so I can now build the pictureflow index on a sane timescale.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: chris_s on May 12, 2022, 11:51:16 AM
Can you post the actual size of the database that's displayed in the Debug menu? Kind of curious how large we are talking here.

Also, if you're interested in experimenting a bit (and don't mind getting into somewhat hacky territory). I could offer up a modification that I'm personally using, where dircache_search (https://github.com/Rockbox/rockbox/blob/7363d65f10ac1b14f6e924ea6b78da355efaa936/firmware/common/dircache.c#L2709) always returns 0 in L2719. In my experience, this makes loading the database into RAM much faster. Which also means you can start browsing the database (or building the PF index) at full speed much earlier – the database is unfortunately extremely slow to access if you use it at the same time as the RAM cache is being loaded.

Would be interesting to see whether this changes anything for you, in terms of getting the database loaded reliably into RAM without weird incantations.

https://www.dropbox.com/s/zuud7swgt168nqu/rockbox-full-ipodclassic-2mib-logf-dircache0.zip?dl=0

It has the disadvantage that you can't retrieve metadata from RAM outside of the database context. That doesn't appear to be a problem in most scenarios though (one of the exceptions being themes with an integrated playlist viewer, which won't show all metadata anymore), especially when using Flash storage.

The alternative is to turn off the dircache completely, but that makes file access very slow in certain scenarios, and is easily perceptible e.g. in PictureFlow.

Full disclosure: My understanding of the dircache is fairly limited at this point so I could be doing something idiotic. Nonetheless, the solution has served me very well so far, so maybe you want to give it a try.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 12, 2022, 04:50:32 PM
It says:

0/17345636 B
 on one of them
 and
0/17519324 B
 on another one

Will try what you suggest and let you know how it goes.

Can I mention again that I'd be very grateful if the increased buffer size can be 'mainlined' somehow, if only for the ipod classics?  Don't want to lose ability to use pictureflow again the next time I update the RB installation.

Thanks again to you (and I presume to "EricLecarde"?) for enhancing this plug-in.  It's significantly improved from what it was.

Ultimately it seems like it would be nice if pictureflow could be integrated into the main browser options, so cover art in various formats could be part of the normal album browser.

E.g. to allow album browsing as on Sony players as here:

https://www.trustedreviews.com/wp-content/uploads/sites/54/2008/11/9237-nwzs630fimg2-1.jpg

or as here:

https://www.wired.com/images_blogs/gadgetlab/2009/09/sony-walkman.jpg

Heh, maybe it did that on the ipod original firmware - it's been so long since I used it I can't remember how it worked.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 12, 2022, 05:00:29 PM
Though even one that says, under debug/view databaseinfo,
 RAM Cache: No
 also says
 RAM: 0/17345636 B

...so I'm not certain if that figure is the actual size of the database.  At any rate, it's what it reports under the debug menu, whatever it means.


Ah - on one that has definitely actually cached the database I see it says 17312616/17345636.  Presumably the figure before the 'oblique' is the space used and the figure after is the size of total RAM available for the cache on the device?  Not sure why that isn't exactly the same on every player.  Or is it Remaining Space/Used Space?
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: chris_s on May 13, 2022, 08:52:46 AM
Wow, that's a huge library. How large is your dircache? (Debug Menu -> View dircache info), and is there enough space left for the playback buffer (System -> Rockbox Info -> Buffer, while something is playing)? You certainly seem to be making good use of the 64MB of RAM ... :)

The second number is what's been allocated in RAM. The allocation is handled by the tagcache thread when it is started (i.e. usually when the device boots). The first number is what's actually used of that allocation, after the tag cache has been successfully loaded into RAM. For example, if you were to already have a large database in RAM, and later re-initialized  the database using only a smaller set of files, it would be able to utilize (a smaller percentage of) the existing allocation. Once you rebooted, the allocation itself would again shrink close to  what's actually needed

I'll be happy to upload a change to Gerrit later today, w.r.t. the plugin buffer size. We'll see if anyone objects or is willing to merge it. :D

I don't disagree that it would be nice to have cover art directly integrated into the database browser. Doesn't seem like a trivial endeavour though...
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 13, 2022, 12:15:45 PM
dircache 6291454B
buffer 34MB

Yeah, I might be coming close to the limits of this 15 year-old tech.

That's without including any of the podcasts or audiobooks in the database (I find it simpler to deal with those via the file browser).
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 13, 2022, 01:22:09 PM
I don't disagree that it would be nice to have cover art directly integrated into the database browser. Doesn't seem like a trivial endeavour though...

Yeah, true.  Unless I can get up to speed so as to work on it myself, I realise this is just pie-in-the-sky and a vague hope, but seems to me the most plausible way to do it would be to start at the other end and gradually expand the functionality of picture-flow to subsume/replicate/replace the functions of the database browser.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Bilgus on May 13, 2022, 03:30:05 PM
Devices use flush callbacks to write data to disk infrequently for harddrives and often for sd drives
I wonder if removing the define for spinning drive would make a difference or if its already taken in to account

 

Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: chris_s on May 13, 2022, 08:21:49 PM
I just noticed the "update cache" function isn't actually working and instead does the  same thing as "rebuild cache".  The "preparing artwork" step should go a *lot* quicker when you've only updated the database, once that's been fixed. The previous stages for building the index will still require the database cache in RAM to work at a reasonable speed though.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 13, 2022, 09:55:07 PM
I've noticed a couple of times now the database seemingly erasing or corrupting itself and asking to be rebuilt.  I'm not sure if it's a real 'thing' or if I've simply gotten confused as to which ipods have already had the DB built, but I'm suspecting it might be a side-effect of having 'load to ram' enabled on iFlash modded ipods.  I vaguely remember that being a problem long ago, and that's why I didn't use 'load to ram' for the DB.  Need to check further what's happening, but may have to just use DB caching when building the pictureflow index, and then turn it off again.

(Just needs a menu option for "load the DB into RAM _right now_ you wretched obstreperous device, and not when you feel like getting round to doing it!")

I can't say I ever noticed any sluggishness accessing the database (for browsing and selecting tracks) without that option enabled.  But for pictureflow indexing it apparently makes a huge difference - seems to mean a ten-fold speed increase in that case.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 14, 2022, 07:52:21 AM
I just noticed the "update cache" function isn't actually working and instead does the  same thing as "rebuild cache".  The "preparing artwork" step should go a *lot* quicker when you've only updated the database, once that's been fixed. The previous stages for building the index will still require the database cache in RAM to work at a reasonable speed though.

Thanks.  If you get that fixed, that will be the next thing to try, I guess - to see if I can get the art index to update after updating the RB database (or adding missing cover art).  The only tricky issue now seems to be how to persuade the device to actually cache the DB when required.  Still can't find a reliable way to trigger it to do that.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: amachronic on May 14, 2022, 10:58:33 AM
The ramcache should kick in at boot time as long as (1) the database is built; and (2) load to RAM is enabled.

The only reason it would not load, as far as I can tell, is if there's an error in load_tagcache(). But most of those errors seem to be related to the database files being corrupted, so perhaps there's an I/O error happening somewhere.

Maybe you can try this build and post the log (system > debug > dump log file) after a failed load to RAM attempt?
https://drive.google.com/file/d/12C2IxlSvvH-AGv-ZS-cqxiQCEXv-d7m2/view?usp=sharing
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 14, 2022, 11:18:39 AM
I'll try that one.  Though when I tried Chris_s's build with logging enabled before, I got log files like this

Code: [Select]
committing tagcache
master file open failed for R/W
commit 83080 entries...
Building index: 0
master file open failed for R/W
tag file open failed: tag=0 write=1 file=/.rockbox/database_0.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
Creating new DB
inserting new tags...
done
sorted 6084 tags
updating indices...
done
updating new indices...
done
s:0/162792/162792
Building index: 1
tag file open failed: tag=1 write=1 file=/.rockbox/database_1.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
inserting new tags...
done
sorted 7214 tags
updating indices...
done
updating new indices...
done
s:1/267032/429824
Building index: 2
tag file open failed: tag=2 write=1 file=/.rockbox/database_2.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
inserting new tags...
done
sorted 478 tags
updating indices...
done
updating new indices...
done
s:2/13120/442944
Building index: 3
tag file open failed: tag=3 write=1 file=/.rockbox/database_3.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
inserting new tags...
done
sorted 83080 tags
updating indices...
done
updating new indices...
done
s:3/2466776/2909720
Building index: 4
tag file open failed: tag=4 write=1 file=/.rockbox/database_4.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
updating new indices...
done
s:4/6836063/2909720
Building index: 5
tag file open failed: tag=5 write=1 file=/.rockbox/database_5.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
inserting new tags...
done
sorted 3484 tags
updating indices...
done
updating new indices...
done
s:5/141784/3051504
Building index: 6
tag file open failed: tag=6 write=1 file=/.rockbox/database_6.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
inserting new tags...
done
sorted 29615 tags
updating indices...
done
updating new indices...
done
s:6/3677784/6729288
Building index: 7
tag file open failed: tag=7 write=1 file=/.rockbox/database_7.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
inserting new tags...
done
sorted 1226 tags
updating indices...
done
updating new indices...
done
s:7/32112/6761400
Building index: 8
tag file open failed: tag=8 write=1 file=/.rockbox/database_8.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
inserting new tags...
done
sorted 41128 tags
updating indices...
done
updating new indices...
done
s:8/1415504/8176904
Building index: 12
tag file open failed: tag=12 write=1 file=/.rockbox/database_12.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
inserting new tags...
done
sorted 6085 tags
updating indices...
done
updating new indices...
done
s:12/162816/8339720
Building numeric indices...
83080/83080 entries processed
tagcache committed
tagcache built!
tagcache check done

Not sure what that signifies.  Are those errors relevant or was it failing to open the files because of existing database files still there?  It specifically never reported anything about loading to ram or even attempting to do so.
When it eventually succeeded I got:


committing tagcache
nothing to commit
tagcache: 17519324 bytes allocated.
loading tagcache to ram...

Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 14, 2022, 11:18:55 AM

Got one more ipod to build the pictureflow index - this time it required making a failed attempt to do so (without caching the database first, so it seemed set to take hours, so I aborted it at 0% progress).  After aborting that attempt it _then_ reported it had now cached the database, so I started the pictureflow build again and this time it worked much faster.  Just rebooting the thing didn't seem to do it, nor did leaving it alone for a few minutes, it seemed to be the first aborted attempt at building pictureflow that persuaded it to then cache the DB.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: amachronic on May 14, 2022, 11:28:30 AM
Those 'errors' in the database build are harmless if it's a fresh database build, it's just trying to check if the file exists and if not it creates it. The part relevant to ramcache is at the end:
Code: [Select]
loading tagcache to ram...
signifies the start of the load to RAM. When it finishes you should see this:
Code: [Select]
tagcache loaded into ram!
utilization: 99%
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 14, 2022, 01:24:02 PM
Hmm OK.  Not sure when I eventually got the 'loading tagcache' entry - that appeared on its own in a later log file.   Tried repeatedly and got lots of log files - most attempts just didn't say anything at all about loading - certainly didn't see any messages reporting a failure at doing so, most times it just didn't seem to even try.

I guess I probably need to try logging again and be more methodical about seeing what it says specifically after rebooting or shutting down and restarting.  (Does it matter whether one reboots via centre-button+menu or shuts down by long-pressing play then restarts?  Is there a difference between the two?)

The most recent attempt gave a whole different error message - despite correctly loading the DB cache to RAM, this time it got to the end of picture flow indexing and then said something about 'unable to load empty slide'.  That might be something hardware-related about that particular ipod, maybe? [Edit] That one worked on the second attempt!
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: chris_s on May 15, 2022, 08:19:40 AM
Pressing the Menu and Center buttons force restarts your iPod, which you’ll want to avoid.

Do you experience any data corruption when using Rockbox’s USB mode? That seems like a litmus test for when to expect issues to pop up in normal operation as well, and may indicate that the combination of iPod classic and the iFlash adapter or SD cards you’re using may be still be causing issues with Rockbox.
Title: Re: pictureflow problem - any suggestions for a fix/workaround?
Post by: Frankenpod on May 15, 2022, 08:59:22 AM
Good question, though I don't really know, as I've always avoided using rockbox's USB mode, precisely because, at least in the early days, it was very likely to cause corruption.  I get the impression that's improved with more recent versions of RB but I've continued to avoid risking finding out.  (Really don't want to risk ending up with a file system corrupted to the point where I have to restore the whole thing completely)

Currently picture flow is working well on multiple ipods.  Will try again installing the logging version and see what's going on with loading DB to RAM.  Never got any actual error messages in the logs about that before, though, it just either did it or, more often, seemed to not even attempt to do so.

Been repairing cover art for all the albums (on the computer) where it's missing or is too large/wrong format to be displayed.  Looking through pictureflow it becomes easy to see all the art that is, for example, in embedded png format or has some other problem.    Once your improved 'update' work is in a dev version I'll have a go with installing it and resyncing then updating the picture DB (and see if it does it in reasonable time).