Playlist trouble with UTF8

[APz]

Many of the files in my iRiver H120 have scandinavian letters (ä, ö and å) in the filenames. They are mostly encoded latin-1, but some are UTF-8. I've noticed, that I can't add files with latin-1 encoded scands in their names. After trying to play those files, they seem to have utf-8 encoded names in the playlist and (ERR) before the filename. Is this a known problem?

[bluebrother]

Many of the files in my iRiver H120 have scandinavian letters (ä, ö and å) in the filenames. They are mostly encoded latin-1, but some are UTF-8.

This is not possible. VFAT uses UCS-2 to encode the filenames, you can't simply use utf8. If you have different encodings for the filenames something is for sure broken with the filenames.

I've noticed, that I can't add files with latin-1 encoded scands in their names. After trying to play those files, they seem to have utf-8 encoded names in the playlist and (ERR) before the filename. Is this a known problem?

Rockbox creates playlists using utf-8. Those playlists get the extension "m3u8". The format is identical to m3u  but it uses utf-8 encoding. If you create playlists on the PC make sure to either create them using utf-8 and use the extension m3u8 or create them using latin-1 and set the codepage setting in Rockbox to latin-1 (and use the file extension m3u).

[APz]

Rockbox creates playlists using utf-8. Those playlists get the extension "m3u8". The format is identical to m3u  but it uses utf-8 encoding. If you create playlists on the PC make sure to either create them using utf-8 and use the extension m3u8 or create them using latin-1 and set the codepage setting in Rockbox to latin-1 (and use the file extension m3u).

I've now set the codepage in Rockbox to Latin1. Still, when I add files to playlist, they won't play and have "(ERR)" before their filenames, and the names show raw utf-8 characters in Rockbox's "view playlist" list.

The interesting part is, that you can play those files if you just select a file from a directory and just play it; but if I add files to the playlist that was generated by the previous action, the newly added files show the raw UTF-8 characters and won't play.

[bluebrother]

I've now set the codepage in Rockbox to Latin1. Still, when I add files to playlist, they won't play and have "(ERR)" before their filenames, and the names show raw utf-8 characters in Rockbox's "view playlist" list.

errr ... how exactly did you create that playlist? If I understand correctly you did that on the player and not the PC or have I understood it wrong?

[APz]

I've now set the codepage in Rockbox to Latin1. Still, when I add files to playlist, they won't play and have "(ERR)" before their filenames, and the names show raw utf-8 characters in Rockbox's "view playlist" list.

errr ... how exactly did you create that playlist? If I understand correctly you did that on the player and not the PC or have I understood it wrong?

I used only the player itself.

Here's a quick way to reproduce the problem:

Have a directory with just couple of files in it, every file having atleast one scandinavian letter in their names.

1) Open the file browser, select one of the songs. Rockbox will ask if I want to lose the dynamic playlist, I hit the navi-button and the song starts to play.
2) I enter the file browser again, select the very same song I selected earlier, hold the navi button on it for a longer time to get the context menu. I then select playlist and insert to playlist.

Now I have a playlist, with the same same song twice, except the first attempt works and the latter won't.

This was the main "WTF"-moment for me

[Lear]

Strange behavior indeed, but I can't reproduce it. Bluebrother is probably right, there's something strange with the filesystem.

What happens if you save the playlist, does it show the same behavior?

In any case, I'd like a copy of such a playlist, as well as the file .rockbox/.playlist_control (as it is after step 2). I might not be able to do anything about it, but it would be interesting to see the files anyway.

[APz]

Many of the files in my iRiver H120 have scandinavian letters (ä, ö and å) in the filenames. They are mostly encoded latin-1, but some are UTF-8.

This is not possible. VFAT uses UCS-2 to encode the filenames, you can't simply use utf8. If you have different encodings for the filenames something is for sure broken with the filenames.

The files were produced by the "abcde" script, and have utf-8 in their filenames. I simply copied them to the iRiver, and doing "ls" shows them correctly, atleast on Linux's side. Browsing the files in Rockbox shows the filenames as "raw" utf-8, but seeing no harm in this (as I'm more interested in the tags, and they do show up correctly) I never bothered to rename anything.

I've attached a "screenshot" of how the filenames show up after being added into a freshly created playlist. They play just fine, until the playlist is saved.

[APz]

Strange behavior indeed, but I can't reproduce it. Bluebrother is probably right, there's something strange with the filesystem.
What happens if you save the playlist, does it show the same behavior?
In any case, I'd like a copy of such a playlist, as well as the file .rockbox/.playlist_control (as it is after step 2). I might not be able to do anything about it, but it would be interesting to see the files anyway.

Adding the the files into a fresh playlist allows playing them, but saving the playlist and then reloading it will cause all the files with UTF-8 in their filenames to have the (ERR) in their filenames.

I've attached a small "screenshot" what happens after the playlist is reloaded, and the playlist itself. The playlist had to be renamed to .txt as the forum didn't accept the original filename :/

[Lear]

Hm, the utf-8-encoded filenames seem to get corrupted when written to (or possibly read from) disk, but I can't really see how. I'd understand it if you saved the playlist with an ".m3u" extension, but that doesn't explain the other case (where inserting a track in the playlist cause the same behavior). Getting a copy of ".rockbox/.playlist_control" from before the playlist has been saved might help...

[markun]

I suspect that the drive is mounted with the wrong codepage settings. Do you mount with " -o iocharset=utf8" when you copy UTF-8 encoded files? I would advise you to have all your filenames in the same encoding btw, makes life easier.

But I also don't understand how the same file ends up in the playlist with 2 different encodings. Sounds like a bug in rockbox to me.

[Lear]

But I also don't understand how the same file ends up in the playlist with 2 different encodings. Sounds like a bug in rockbox to me.

In this case, the two files are added using very different code paths. When starting playback, the code is simple, the name is just copied to a buffer. When inserting a new track however, things get more complicated, involving writing it to the playlist control file. Don't really see how that could affect things though.