pictureflow problem - any suggestions for a fix/workaround?

[Frankenpod]

I don't disagree that it would be nice to have cover art directly integrated into the database browser. Doesn't seem like a trivial endeavour though...

Yeah, true.  Unless I can get up to speed so as to work on it myself, I realise this is just pie-in-the-sky and a vague hope, but seems to me the most plausible way to do it would be to start at the other end and gradually expand the functionality of picture-flow to subsume/replicate/replace the functions of the database browser.

[Bilgus]

Devices use flush callbacks to write data to disk infrequently for harddrives and often for sd drives
I wonder if removing the define for spinning drive would make a difference or if its already taken in to account

 

[chris_s]

I just noticed the "update cache" function isn't actually working and instead does the  same thing as "rebuild cache".  The "preparing artwork" step should go a *lot* quicker when you've only updated the database, once that's been fixed. The previous stages for building the index will still require the database cache in RAM to work at a reasonable speed though.

[Frankenpod]

I've noticed a couple of times now the database seemingly erasing or corrupting itself and asking to be rebuilt.  I'm not sure if it's a real 'thing' or if I've simply gotten confused as to which ipods have already had the DB built, but I'm suspecting it might be a side-effect of having 'load to ram' enabled on iFlash modded ipods.  I vaguely remember that being a problem long ago, and that's why I didn't use 'load to ram' for the DB.  Need to check further what's happening, but may have to just use DB caching when building the pictureflow index, and then turn it off again.

(Just needs a menu option for "load the DB into RAM _right now_ you wretched obstreperous device, and not when you feel like getting round to doing it!")

I can't say I ever noticed any sluggishness accessing the database (for browsing and selecting tracks) without that option enabled.  But for pictureflow indexing it apparently makes a huge difference - seems to mean a ten-fold speed increase in that case.

[Frankenpod]

I just noticed the "update cache" function isn't actually working and instead does the  same thing as "rebuild cache".  The "preparing artwork" step should go a *lot* quicker when you've only updated the database, once that's been fixed. The previous stages for building the index will still require the database cache in RAM to work at a reasonable speed though.

Thanks.  If you get that fixed, that will be the next thing to try, I guess - to see if I can get the art index to update after updating the RB database (or adding missing cover art).  The only tricky issue now seems to be how to persuade the device to actually cache the DB when required.  Still can't find a reliable way to trigger it to do that.

[amachronic]

The ramcache should kick in at boot time as long as (1) the database is built; and (2) load to RAM is enabled.

The only reason it would not load, as far as I can tell, is if there's an error in load_tagcache(). But most of those errors seem to be related to the database files being corrupted, so perhaps there's an I/O error happening somewhere.

Maybe you can try this build and post the log (system > debug > dump log file) after a failed load to RAM attempt?
https://drive.google.com/file/d/12C2IxlSvvH-AGv-ZS-cqxiQCEXv-d7m2/view?usp=sharing

[Frankenpod]

I'll try that one.  Though when I tried Chris_s's build with logging enabled before, I got log files like this

Code: [Select]

committing tagcache
master file open failed for R/W
commit 83080 entries...
Building index: 0
master file open failed for R/W
tag file open failed: tag=0 write=1 file=/.rockbox/database_0.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
Creating new DB
inserting new tags...
done
sorted 6084 tags
updating indices...
done
updating new indices...
done
s:0/162792/162792
Building index: 1
tag file open failed: tag=1 write=1 file=/.rockbox/database_1.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
inserting new tags...
done
sorted 7214 tags
updating indices...
done
updating new indices...
done
s:1/267032/429824
Building index: 2
tag file open failed: tag=2 write=1 file=/.rockbox/database_2.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
inserting new tags...
done
sorted 478 tags
updating indices...
done
updating new indices...
done
s:2/13120/442944
Building index: 3
tag file open failed: tag=3 write=1 file=/.rockbox/database_3.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
inserting new tags...
done
sorted 83080 tags
updating indices...
done
updating new indices...
done
s:3/2466776/2909720
Building index: 4
tag file open failed: tag=4 write=1 file=/.rockbox/database_4.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
updating new indices...
done
s:4/6836063/2909720
Building index: 5
tag file open failed: tag=5 write=1 file=/.rockbox/database_5.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
inserting new tags...
done
sorted 3484 tags
updating indices...
done
updating new indices...
done
s:5/141784/3051504
Building index: 6
tag file open failed: tag=6 write=1 file=/.rockbox/database_6.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
inserting new tags...
done
sorted 29615 tags
updating indices...
done
updating new indices...
done
s:6/3677784/6729288
Building index: 7
tag file open failed: tag=7 write=1 file=/.rockbox/database_7.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
inserting new tags...
done
sorted 1226 tags
updating indices...
done
updating new indices...
done
s:7/32112/6761400
Building index: 8
tag file open failed: tag=8 write=1 file=/.rockbox/database_8.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
inserting new tags...
done
sorted 41128 tags
updating indices...
done
updating new indices...
done
s:8/1415504/8176904
Building index: 12
tag file open failed: tag=12 write=1 file=/.rockbox/database_12.tcd
lookup_buffer_depth=83082
commit_entry_count=83081
Loading index file
inserting new tags...
done
sorted 6085 tags
updating indices...
done
updating new indices...
done
s:12/162816/8339720
Building numeric indices...
83080/83080 entries processed
tagcache committed
tagcache built!
tagcache check done

Not sure what that signifies.  Are those errors relevant or was it failing to open the files because of existing database files still there?  It specifically never reported anything about loading to ram or even attempting to do so.
When it eventually succeeded I got:


committing tagcache
nothing to commit
tagcache: 17519324 bytes allocated.
loading tagcache to ram...

[Frankenpod]

Got one more ipod to build the pictureflow index - this time it required making a failed attempt to do so (without caching the database first, so it seemed set to take hours, so I aborted it at 0% progress).  After aborting that attempt it _then_ reported it had now cached the database, so I started the pictureflow build again and this time it worked much faster.  Just rebooting the thing didn't seem to do it, nor did leaving it alone for a few minutes, it seemed to be the first aborted attempt at building pictureflow that persuaded it to then cache the DB.

[amachronic]

Those 'errors' in the database build are harmless if it's a fresh database build, it's just trying to check if the file exists and if not it creates it. The part relevant to ramcache is at the end:

Code: [Select]

loading tagcache to ram...
signifies the start of the load to RAM. When it finishes you should see this:

Code: [Select]

tagcache loaded into ram!
utilization: 99%


[Frankenpod]

Hmm OK.  Not sure when I eventually got the 'loading tagcache' entry - that appeared on its own in a later log file.   Tried repeatedly and got lots of log files - most attempts just didn't say anything at all about loading - certainly didn't see any messages reporting a failure at doing so, most times it just didn't seem to even try.

I guess I probably need to try logging again and be more methodical about seeing what it says specifically after rebooting or shutting down and restarting.  (Does it matter whether one reboots via centre-button+menu or shuts down by long-pressing play then restarts?  Is there a difference between the two?)

The most recent attempt gave a whole different error message - despite correctly loading the DB cache to RAM, this time it got to the end of picture flow indexing and then said something about 'unable to load empty slide'.  That might be something hardware-related about that particular ipod, maybe? [Edit] That one worked on the second attempt!

[chris_s]

Pressing the Menu and Center buttons force restarts your iPod, which you’ll want to avoid.

Do you experience any data corruption when using Rockbox’s USB mode? That seems like a litmus test for when to expect issues to pop up in normal operation as well, and may indicate that the combination of iPod classic and the iFlash adapter or SD cards you’re using may be still be causing issues with Rockbox.

[Frankenpod]

Good question, though I don't really know, as I've always avoided using rockbox's USB mode, precisely because, at least in the early days, it was very likely to cause corruption.  I get the impression that's improved with more recent versions of RB but I've continued to avoid risking finding out.  (Really don't want to risk ending up with a file system corrupted to the point where I have to restore the whole thing completely)

Currently picture flow is working well on multiple ipods.  Will try again installing the logging version and see what's going on with loading DB to RAM.  Never got any actual error messages in the logs about that before, though, it just either did it or, more often, seemed to not even attempt to do so.

Been repairing cover art for all the albums (on the computer) where it's missing or is too large/wrong format to be displayed.  Looking through pictureflow it becomes easy to see all the art that is, for example, in embedded png format or has some other problem.    Once your improved 'update' work is in a dev version I'll have a go with installing it and resyncing then updating the picture DB (and see if it does it in reasonable time).

[Frankenpod]

Really noticeable that it makes an order-of-magnitude difference to the speed of building pictureflow indexes if the database is actually cached to RAM.  Still can't quite work out what persuades it to actually do that.  Sometimes it's just a case of leaving it alone for a while (during which time the disk-activity indicator is shown - maybe it just takes a long time to read the whole thing from the flash storage?)

I haven't been able to test the revamped "update cover art" option because until/unless the increased buffer-size patch gets merged the later dev versions won't work for me due to that running out of memory issue.  Unless I can get dev environment working again and build my own patched version, of course.

[chris_s]

Really noticeable that it makes an order-of-magnitude difference to the speed of building pictureflow indexes if the database is actually cached to RAM.  Still can't quite work out what persuades it to actually do that.  Sometimes it's just a case of leaving it alone for a while (during which time the disk-activity indicator is shown - maybe it just takes a long time to read the whole thing from the flash storage?)

I haven't been able to test the revamped "update cover art" option because until/unless the increased buffer-size patch gets merged the later dev versions won't work for me due to that running out of memory issue.  Unless I can get dev environment working again and build my own patched version, of course.

to tide you over... the latest dev build with a 2mib plugin buffer: https://www.dropbox.com/s/8at7v4jzd10webm/rockbox-full-ipodclassic-2mib-plugin.zip?dl=0

With a database of your size, I would expect at least a few minutes of waiting time until everything's loaded into RAM after booting (with both dircache and tagcache in RAM) – and possibly much longer – which is what led me to the "solution" I mentioned in one of my previous posts, which speeds up that process quite a bit. Maybe I can turn that into an official option at some point: https://forums.rockbox.org/index.php/topic,54031.msg250401.html#msg250401

[Frankenpod]

Thanks again, that's very helpful.  Been experimenting with that build - does certainly _seem_ as if it makes the picture-flow build process faster still, but it's hard to quantify for sure, because there still seem to be a great number of (I suspect iflash-mod-related) glitches to deal with before getting to that point, so I didn't get to time it methodically.  (I feel like, once database was built and loaded to RAM, it took about 10-20 minutes for the whole picture-flow prep process, rather than the 30 minutes it took previously, but forgot to note the time it started!)

  Had problems updating the database to start with (before could even get to picture flow) because - and it seems as if having 'load to RAM' set to on is related to the problem - the database has an annoying tendency to take ages to build, and then mysteriously disappear and have to be rebuilt again - particularly after a shut-down/restart.  Plus there's the usual issue of getting it to actually load into RAM (seems like it requires a shut-down/restart and then a long wait while keeping an eye on the disk-activity icon..._if_ the database doesn't disappear entirely with the shut-down)[Edit - nope, seems even that doesn't do it reliably...sometimes it does, other times no matter how long you leave it the thing just sits there and makes no effort to actually load the DB].

Once working though it's quite impressive being able to flick through so many albums of cover art.