Rockbox Technical Forums

Support and General Use => Audio Playback, Database and Playlists => Topic started by: Warphine on November 11, 2019, 12:00:57 AM

Title: Updating Database Never Finishes
Post by: Warphine on November 11, 2019, 12:00:57 AM
Hello! I have a 1TB iPod Classic (7th Gen) modded with the iFlash Dual SD adapter (2x 512 GB) and I recently hit 50,000 songs and the original OS just keeps crashing. I knew this day would come so this weekend I installed Rockbox and began to learn how to use it.

I thought I had everything good to go, but today I tried adding a new CD to my iPod using iTunes and the original iPod OS. It synced it, then I disconnected and restarted the iPod into Rockbox and updated the database. It said it was scanning in the background, and then I noticed the iPod get really sluggish. Eventually, I saw the new CD show up in the list but the iPod was still sluggish so I restarted it. This time, it booted up and said it was doing something to the Database. It got to 2/9 and booted into the main menu. This time it was responding well. I entered the Database section and it said it needed to reinitialize the database so I let it. It started scanning for files and eventually it got to 50,499 (or so) files and just stayed there for over an hour. Rebooting it from here, just causes it to have to reinit the database from scratch again to which it seemingly never completes.

This is the second time I've added something to the iPod this weekend and had this issue. I've even tried uninstalling and reinstalling Rockbox to try to fix it to no avail.

Am I doomed to not be able to use iTunes to sync music with this thing? I'd rather be able to continue to use iTunes instead of the drag/drop method but I'll switch if I have to.

Prior to adding these CDs, the database was fine and I was listening to music on it without any issues.

Thanks for the help!
Title: Re: Updating Database Never Finishes
Post by: saratoga on November 11, 2019, 12:54:27 AM
Flash modded 6th gens tend to work badly with our drivers, so instability when scanning the disk is often reported. You should probably copy files with the Apple firmware and stick to the file browser.
Title: Re: Updating Database Never Finishes
Post by: Frankenpod on November 11, 2019, 06:22:09 AM
Of course, you won't be able to use the file-browser either, because you are using iTunes to sync, and it obfuscates file names.

Why are you syncing with iTunes?  Be better to just copy the files directly, keeping the proper file names and directory structure.


Have you tried deleting the existing failed database files before you have another go at rebuilding?  Sometimes (with the flash adaptor problem) it will succeed at building the database on a subsequent attempt if you completely remove all traces of the failed attempt.
Title: Re: Updating Database Never Finishes
Post by: Warphine on November 11, 2019, 10:04:10 AM
I like having the iTunes GUI when playing music on the computer and the smart playlist ability is quite good. I suppose I could just keep the iTunes library updated manually while also dragging the music to Rockbox in the future. I'd just need to clear out all the music on it and start over.

If I want to try deleting the database, I'm assuming that involves going into the .rockbox folder and deleting all the files named database? I tried that but when it restarts it says "Committing database 1/9" and gets stuck there.

Thanks everyone for taking the time to respond.
Title: Re: Updating Database Never Finishes
Post by: Frankenpod on November 11, 2019, 01:18:51 PM
If it goes straight to 'committing database 1/9' then you haven't deleted all the relevant files, because without any database files there, it should go back to the first step of going through all the tracks and building the database.  You deleted everything *.tcd?  And those files definitely were gone (I'm wondering if there are real disk access problems, perhaps it says they are deleted but they actually stay there?).

  Of course, it might build the database _then_ go back to getting stuck on the commit phase, but it should at least do the previous step again first.

If you can't get the database to work, the only option is to give up on iTunes so you can use the file browser instead.


Oh, though a final option is to use the simulator to build/update the database on the computer, then copy the result to the ipod (that's what I do now, in fact).

What brand of sd card are you using?  PNY work better than most, I find (in fact I'm trying to get people to report what luck they have with different brands, because PNY are now expensive to get - I know Kingston aren't great...I mean, really the problem is the iFlash and Rockbox not playing nicely together, but the effect of it seems to vary depending on how tolerant a given card brand is).
Title: Re: Updating Database Never Finishes
Post by: Warphine on November 11, 2019, 02:02:46 PM
I'm going on a road trip tomorrow and don't want to be without my iPod so I basically just reformatted it and now I'm pushing all my music to it the file browser way. I didn't know about the simulator. Would have been good to test that route out. I can always reformat it this weekend and give that a try. Just takes like 12-20 hours to push all the music to this thing haha.

As far as SD card types, I use PNY. These ones in fact:

https://www.amazon.com/gp/product/B00XJRX01M/ref=ppx_yo_dt_b_asin_title_o01_s00?ie=UTF8&psc=1

Thanks again!
Title: Re: Updating Database Never Finishes
Post by: cereal_killer on November 11, 2019, 02:12:18 PM
Don't forget to use Apple disk mode to transfer files to avoid file corruption.
Title: Re: Updating Database Never Finishes
Post by: Frankenpod on November 11, 2019, 02:41:35 PM
I'm going on a road trip tomorrow and don't want to be without my iPod so I basically just reformatted it and now I'm pushing all my music to it the file browser way. I didn't know about the simulator. Would have been good to test that route out. I can always reformat it this weekend and give that a try. Just takes like 12-20 hours to push all the music to this thing haha.

As far as SD card types, I use PNY. These ones in fact:

https://www.amazon.com/gp/product/B00XJRX01M/ref=ppx_yo_dt_b_asin_title_o01_s00?ie=UTF8&psc=1

Thanks again!

That's what I use, but PNY cards are far more costly than other brands in UK, though.  Really curious to know if any cheaper brands work OK, but nobody seems to know (as I say, I didn't find Kingston were much good in this respect).
Title: Re: Updating Database Never Finishes
Post by: Warphine on November 11, 2019, 06:24:09 PM
Don't forget to use Apple disk mode to transfer files to avoid file corruption.

Thanks for the tip. Unfortunately, I didn't know that and I'm currently transferring files while in Rockbox's USB Keypad Mode screen. I guess if there are issues I'll just have to reformat and start over.
Title: Re: Updating Database Never Finishes
Post by: Warphine on November 11, 2019, 06:40:14 PM
Oh, though a final option is to use the simulator to build/update the database on the computer, then copy the result to the ipod (that's what I do now, in fact).

Any chance you got a link to this simulator app so I can check it out? Thanks!
Title: Re: Updating Database Never Finishes
Post by: Frankenpod on November 11, 2019, 08:00:02 PM
Up there to the top left - downloads - extras - simulators!
Title: Re: Updating Database Never Finishes
Post by: Warphine on November 12, 2019, 10:09:15 AM
Just finished syncing my files to the iPod using file browser and seems to be working great. Unfortunately though, I just realized that my old playlists won't work anymore as they are probably tied to the old iTunes 4 letter file names.

This will be fine for now, but I might want to try reformatting it this weekend and giving your simulator a try to build the database (which makes sense as my PC is way more powerful and should be a lot faster doing that on the PC than on the iPod itself) and see if that gets it to work for me. Then I can use the Rockbox Playlist Converter to get my playlists to work properly again (they were working before).

Any tips on setting up/using the simulator?
Title: Re: Updating Database Never Finishes
Post by: cereal_killer on November 12, 2019, 12:03:42 PM
As far as I know, to use the Simulator, the files must be inside the simdisk directory. Otherwise the files are not shown and creating the database won't work. If you are on Linux you could use the database tool you have to compile. Maybe this tool also helps: http://forums.rockbox.org/index.php?topic=20964.0
Title: Re: Updating Database Never Finishes
Post by: Frankenpod on November 12, 2019, 02:19:25 PM
The files don't have to be inside the simdisk directory, but you have to make a hard-link in there to the folder they are actually in.  You'll have to google windows hardlinks, I haven't the energy to explain it!

Also it's surprising how long it takes the PC to fully build the database.  Clearly the simulator isn't doing it very efficiently.  But once built you can use the update option which isn't too bad.

Most annoying thing, though, is that because of how windows file systems and the simulator handle DST, the database has to be rebuilt when the clocks go back or forward!  It's not the only instance when the clock-changing causes problems for computers (something to do with different file systems handling time differently, I think).

Another reason I wish they'd stop messing with the clocks like that!  It's always gotten on my nerves!
Title: Re: Updating Database Never Finishes
Post by: Warphine on November 13, 2019, 04:44:17 PM
The files don't have to be inside the simdisk directory, but you have to make a hard-link in there to the folder they are actually in.  You'll have to google windows hardlinks, I haven't the energy to explain it!

Good thing I'm a "professional" programmer and already know about those things. Thanks man!
Title: Re: Updating Database Never Finishes
Post by: Warphine on November 13, 2019, 04:50:46 PM
So, I got a symbolic link in the simdisk directory. I assume then I go to Database and update it there. It rescans all the files and creates a new database file. Then I just simply copy it to the .rockbox folder on the iPod itself?
Title: Re: Updating Database Never Finishes
Post by: Frankenpod on November 14, 2019, 09:55:49 AM
It creates multiple database files (actually I don't know if you need all of them, but I just copy them all over).

And of course it's crucial that the folder structure on the ipod is the same as that in your library.

Getting playlists working involves a bit of thought, but I'm sure you can work that out for yourself.

(the lack of proper playlist management tools on both iTunes and Media Monkey is a pet peeve).
Title: Re: Updating Database Never Finishes
Post by: Warphine on November 15, 2019, 02:45:47 AM
I managed to get the database updating successfully with the simulator so thank you for your info!

That said, I've have been doing a full Initialize Now pass which takes about 25 minutes (would be nice if this thing could go faster by utilizing more cpu cores or something). Going forward, I'm hoping the Update Now pass is all I need but I haven't tried it yet.

I also noticed that the album art is rarely working. Sometimes it'll show but more often than not, it won't. Is that normal? In case it helps, I embed all my album art images into the mp3's themselves.

Getting playlists working involves a bit of thought, but I'm sure you can work that out for yourself.

I found a converter program that'll get the playlists to work now that I'm back on the iTunes/iPod syncing method.

I would love to somehow write a script that automates all these steps I'm going to have to repeat every time I want to update my iPod.

BTW, I love your name "Frankenpod" as that is what I called my modded iPod for many years. :)
Title: Re: Updating Database Never Finishes
Post by: Frankenpod on November 15, 2019, 11:16:48 AM
Album art works, but rockbox is very fussy about the format.  I used to have a lot of .png embedded art (as that's what iTunes puts in there) but rockbox only sees .jpgs.  And (I've seen mentioned here) it doesn't like progressive jpgs either (never had that issue come up myself).

I guess that's yet another thing it would be nice if someone smart enough could add to rockbox - the ability to use embedded art in formats other than jpg.  Till then you'll have to find a way to bulk convert it all to jpg.  Ages ago I used some script or other with some bulk art conversion utility and tagging program.