pictureflow problem - any suggestions for a fix/workaround?

[Frankenpod]

Yeah, I am convinced part of the problem is the usual glitches with flash-mods and writing to disk.  It seems to work better with the micro-SD-card/Quad adaptor ones.  The ones that use full size cards I find I'll go through the whole database build, the database will be there and accessible, then I'll set 'load to RAM', then when I go back to the database again it's disappeared and insists on rebuilding itself entirely!  That sort of thing was why I used to build the database with the simulator on the PC.  Problem now is I'm not sure the simulators for download use an up-to-date version of RB.

Had more luck with the quad-adaptor ipods (managed to get pictureflow working perfectly well on two of them now), though still haven't worked out what it is that causes the database caching to actually kick in.  It seems rather random.  Definitely seems it makes a big difference to how long the cover art indexing takes, so I'll just have to keep checking the debug menu to see when the cache has actually started doing its thing.

In terms of what the underlying code actually says, when you select 'load database to RAM', at what point does it in fact_do_ that?   Is it supposed to do it immediately you select the option, or is it deferred in some way, till something triggers it to happen?

[Frankenpod]

 It may also sometimes be necessary to restart the device after building the database for the changes to be committed.

For whatever reason, neither rebooting (i.e. holding centre button+menu) nor restarting (holding play till it shuts down then starting up again) seems to reliably cause that 'database cache' to activate.  I don't really know _how_ I got it to work (twice).  It's proving quite mysterious.  (Perhaps it's a hardware issue I'm having?).  But certainly it makes a difference to the time it takes to build the art database.   In addition, the increase you made to the plug-in buffer size seems to be essential to be able to build it at all.

[chris_s]

Yeah, I am convinced part of the problem is the usual glitches with flash-mods and writing to disk.

It's possible, although, at least in my experience with an iPod video and iPod 4G, each with an iFlash Solo and 512GB SD card, they've been rock-solid ever since speachy et al improved the compatibility of the power management handling – ymmv.

In terms of what the underlying code actually says, when you select 'load database to RAM', at what point does it in fact_do_ that?   Is it supposed to do it immediately you select the option, or is it deferred in some way, till something triggers it to happen?

Changing the option doesn't seem to trigger anything (e.g. if the database was in RAM before, it will still be loaded after you've turned the setting off, at least until you update the database or reboot). From quick peeks at the code, I would say it is required to reboot to make sure sufficient space for the database RAM cache has been allocated and can be utilized.

 It may also sometimes be necessary to restart the device after building the database for the changes to be committed.

For whatever reason, neither rebooting (i.e. holding centre button+menu) nor restarting (holding play till it shuts down then starting up again) seems to reliably cause that 'database cache' to activate. 

Yeah, that part seems weird. Just to mention it though, and sorry if this is very obvious, but you will have to wait the same amount of time for the database to be loaded into RAM each time your player boots, even if the setting hasn't changed. As I mentioned before, the way the disk cache and database cache interact, means it can take a while for the combination of the two to load – certainly much longer than it takes for the two to load separately. I usually wait for the HD indicator (if your theme displays one) to disappear to know it's now "safe" to use the DB.

[Frankenpod]

Well, I'm getting there - now have four (quad) flash ipods with pictureflow working.  Trouble is I might lose the pictureflow capacity next time I resync them and have to do it again.  Still can't work out what it is that actually causes the database to be loaded to RAM. 

It seems that maybe it happens if you browse the database and scroll through the whole thing...not sure.

Also, it would be handy if one could set the "context menu plug-in" using the fixed.cfg file, so it would be pictureflow by default.  Some things, like battery capacity and file-browser max items, I found helpful to copy from config.cfg to fixed.cfg, but the selected context menu plug-in doesn't seem to get saved in config.cfg so I can't copy the entry into fixed.cfg, and hence have to reset the plug-in to pictureflow each reboot.

[chris_s]

In theory you shouldn't have to do anything but wait (after rebooting) for the cache to load – no idea what's going on there. 

Re fixed.cfg – I'm only barely aware of that functionality. Seems useful, but shouldn't it still remember the context plugin after a reboot, even if some other setting get reset to the values specified in fixed.cfg?

[Frankenpod]

In theory you shouldn't have to do anything but wait (after rebooting) for the cache to load – no idea what's going on there. 

Re fixed.cfg – I'm only barely aware of that functionality. Seems useful, but shouldn't it still remember the context plugin after a reboot, even if some other setting get reset to the values specified in fixed.cfg?

Yeah, for whatever reason it doesn't seem to remember the context menu plugin.  Other settings get saved in config.cfg or can, more permanently, be stored in fixed.cfg.  I resorted to using fixed.cfg to keep the max file browser entries at 9000, because it otherwise seemed to keep reseting it to 1000 (nowhere near large enough!).  Don't know if that's expected RB behaviour or if it's another iflash glitch.

Can't find information on where the context menu plug-in is recorded - would have expected it to be in config.cfg, but it doesn't seem to be.

Edit - ah, seems it's in rockbox/rocks/plugin.dat.  Or at least it is on a HD-based ipod.  Perhaps it's yet another issue with writing to flash drives?  Nope, it's there on the iflash ipods also, but for some reason RB doesn't seem to be paying attention to it unless I reset it again.

[Frankenpod]

Weird - now it's started remembering the context plug-in, no longer insisting I set it again every reboot.  I give up trying to understand what's going on!

[Bilgus]

I still have some work to do with the openplugins module its independent of the cfg file but is keyed to langids so when the lang file changes it breaks the core ones its nothing major or dangerous just annoying but I haven't decided on a better way yet probably just an export function and better versioning
Id hash language stings but it breaks when switching languages then.

Setting the context plugin is immediate but it won't get saved permanantely unless you do a safe shutdown or leave it on long enough that the device decides to flush to disk and it saves to a temp file first like the cfg files do on flash devices

[Frankenpod]

I still have some work to do with the openplugins module its independent of the cfg file but is keyed to langids so when the lang file changes it breaks the core ones its nothing major or dangerous just annoying but I haven't decided on a better way yet probably just an export function and better versioning
Id hash language stings but it breaks when switching languages then.

Setting the context plugin is immediate but it won't get saved permanantely unless you do a safe shutdown or leave it on long enough that the device decides to flush to disk and it saves to a temp file first like the cfg files do on flash devices

To be honest, don't think I entirely understand all that, but it sounds as if that explains why sometimes the device remembers what plug-in I previously set it to use (picture flow) and other times it seems to forget so I have to set it again.

I think ideally the context menu would actually say the name of the currently selected context-menu plug-in rather than 'open plug in', incidentally.

[Edit] That's not a complaint or anything.  I'm just a bit less baffled now.

[Frankenpod]

Yes! Finally managed to persuade the last iflash ipod to enable database cache.  It definitely makes a huge difference to the speed of building pictureflow index.

But I still haven't manage to work out what it is that causes database caching to suddenly become active.  Have yet to find a pattern to it.

  Seems to be the procedure is: 'turn on load database to RAM, try browsing the database repeatedly, reboot a few times, check debug menu to find it's still not caching the database, browse the database from start to end in hope of persauding it to start using the cache, to no avail, try building pictureflow index and abort it again because it's so painfully slow, give up and leave it alone, come back to it, check to find it's still not caching the DB, reboot it a couple more times...and discover its now finally decided to cache the database, so I can now build the pictureflow index on a sane timescale.

[chris_s]

Can you post the actual size of the database that's displayed in the Debug menu? Kind of curious how large we are talking here.

Also, if you're interested in experimenting a bit (and don't mind getting into somewhat hacky territory). I could offer up a modification that I'm personally using, where dircache_search always returns 0 in L2719. In my experience, this makes loading the database into RAM much faster. Which also means you can start browsing the database (or building the PF index) at full speed much earlier – the database is unfortunately extremely slow to access if you use it at the same time as the RAM cache is being loaded.

Would be interesting to see whether this changes anything for you, in terms of getting the database loaded reliably into RAM without weird incantations.

https://www.dropbox.com/s/zuud7swgt168nqu/rockbox-full-ipodclassic-2mib-logf-dircache0.zip?dl=0

It has the disadvantage that you can't retrieve metadata from RAM outside of the database context. That doesn't appear to be a problem in most scenarios though (one of the exceptions being themes with an integrated playlist viewer, which won't show all metadata anymore), especially when using Flash storage.

The alternative is to turn off the dircache completely, but that makes file access very slow in certain scenarios, and is easily perceptible e.g. in PictureFlow.

Full disclosure: My understanding of the dircache is fairly limited at this point so I could be doing something idiotic. Nonetheless, the solution has served me very well so far, so maybe you want to give it a try.

[Frankenpod]

It says:

0/17345636 B
 on one of them
 and
0/17519324 B
 on another one

Will try what you suggest and let you know how it goes.

Can I mention again that I'd be very grateful if the increased buffer size can be 'mainlined' somehow, if only for the ipod classics?  Don't want to lose ability to use pictureflow again the next time I update the RB installation.

Thanks again to you (and I presume to "EricLecarde"?) for enhancing this plug-in.  It's significantly improved from what it was.

Ultimately it seems like it would be nice if pictureflow could be integrated into the main browser options, so cover art in various formats could be part of the normal album browser.

E.g. to allow album browsing as on Sony players as here:

https://www.trustedreviews.com/wp-content/uploads/sites/54/2008/11/9237-nwzs630fimg2-1.jpg

or as here:

https://www.wired.com/images_blogs/gadgetlab/2009/09/sony-walkman.jpg

Heh, maybe it did that on the ipod original firmware - it's been so long since I used it I can't remember how it worked.

[Frankenpod]

Though even one that says, under debug/view databaseinfo,
 RAM Cache: No
 also says
 RAM: 0/17345636 B

...so I'm not certain if that figure is the actual size of the database.  At any rate, it's what it reports under the debug menu, whatever it means.


Ah - on one that has definitely actually cached the database I see it says 17312616/17345636.  Presumably the figure before the 'oblique' is the space used and the figure after is the size of total RAM available for the cache on the device?  Not sure why that isn't exactly the same on every player.  Or is it Remaining Space/Used Space?

[chris_s]

Wow, that's a huge library. How large is your dircache? (Debug Menu -> View dircache info), and is there enough space left for the playback buffer (System -> Rockbox Info -> Buffer, while something is playing)? You certainly seem to be making good use of the 64MB of RAM ...

The second number is what's been allocated in RAM. The allocation is handled by the tagcache thread when it is started (i.e. usually when the device boots). The first number is what's actually used of that allocation, after the tag cache has been successfully loaded into RAM. For example, if you were to already have a large database in RAM, and later re-initialized  the database using only a smaller set of files, it would be able to utilize (a smaller percentage of) the existing allocation. Once you rebooted, the allocation itself would again shrink close to  what's actually needed

I'll be happy to upload a change to Gerrit later today, w.r.t. the plugin buffer size. We'll see if anyone objects or is willing to merge it.

I don't disagree that it would be nice to have cover art directly integrated into the database browser. Doesn't seem like a trivial endeavour though...

[Frankenpod]

dircache 6291454B
buffer 34MB

Yeah, I might be coming close to the limits of this 15 year-old tech.

That's without including any of the podcasts or audiobooks in the database (I find it simpler to deal with those via the file browser).