File Browser not showing characters like ? and “

[Pommes]

Dear Forum,
I am using the file browser, but for some reason characters like ? and “ show as a square symbol.
Characters like -!,’ á work well and show right.
In playing screen all characters are shown well without any squares.
Can anybody explain this to me?
It is important, because the files and folders with the square do not queue from external generated m3u8, because rockbox does not find them. All other songs play well from these m3u8.
I transferred all the folders and files from Mac to ipod.

Thank you 

[Frankenpod]

Are you saying the tracks have those characters in their file name or just in the track name in the tags?

On windows those characters are specifically not allowed in file names, so this doesn't arise for me.

Is it something to do with the use of FAT32?  I don't know how it works, but does anything using FAT32 necessarily share the same limitations on file-names as windows itself?

[Frankenpod]

If it's a FAT32 limitation you may just have to rename those tracks to remove those particular characters.

[bluebrother]

Is it something to do with the use of FAT32?  I don't know how it works, but does anything using FAT32 necessarily share the same limitations on file-names as windows itself?

Yes.

While you theoretically could simply write them into the byte sequence holding the filename on disk this would violate the FAT spec, and as such cause problems on various systems. So you can't.

I don't know what MacOS is doing exactly here, but if you have those characters in filenames it has to change them when copying those file to a FAT volume. I wouldn't be surprised if MacOS replaced it with a different Unicode character that looks identical, and also did that on its own APFS / HFS+. For ? this could be U+FF1F: https://unicode-table.com/en/FF1F/ Since this is not a ? character (and rarely used) Rockbox fonts are unlikely to support that.

The safest thing is to generally not used such characters in filenames. They end up causing problems in other places too.

[Pommes]

Thank you very much, this answered my question perfectly.
To solve my problem I replaced the following characters
/ ? < > \ : * | " with this character  -   in the tags itself.
After that I used DBpoweramp to rearrange all music.

Some days ago I ran into another problem:
Mac OS uses utf8 Mac encoding, which is different to regular utf 8.
For example á in utf8 Mac is actually 2 characters, while in regular utf 8 it is one character.
So RockBox did not play those files, because my playlists are generated under macOS.

A friend of mine wrote a Mac application, which transforms the playlist from UTF-8 Mac to UTF-8.
He also included a simple find and replace mechanism into the app, so I can modify the paths in the playlist.

I asked him, and he is willing to share it here in this community.
Is anybody interested, and is it allowed to post a link to this app, maybe somebody could host it here on this RockBox forum?
Please let me know.
Thank you very much

[bluebrother]

To solve my problem I replaced the following characters
/ ? < > \ : * | " with this character  -   in the tags itself.
After that I used DBpoweramp to rearrange all music.

You're aware that the problem with those characters only applies to filenames, not to the tags themselves? I do have a couple of files that contain ? in the tag, but they are stripped from the filename.

Also, I had a chance to try it on a Mac. Turns out Mac appears to be using ? (https://unicode-table.com/en/003F/), at least when I have it show the filename in Terminal. That's on the Mac's internal disk. When copying the file to a FAT volume the same applies. However, when I plug that FAT volume into a Linux machine the ? is a different character: it renders as headphone symbol, but when pasting it into Google it's indicated to be in the Unicode Private Use Area. Haven't looked at the raw byte sequence on the volume, but one thing is clear: MacOS uses a different character to store the ?, at least on FAT. Maybe it does so on its own volume as well but internally translates it before showing the file. So pretty much to my initial guess, but it appears to use a different codepoint.

Mac OS uses utf8 Mac encoding, which is different to regular utf 8.
For example á in utf8 Mac is actually 2 characters, while in regular utf 8 it is one character.
So RockBox did not play those files, because my playlists are generated under macOS.

Interesting, didn't knew this was a thing: https://apple.stackexchange.com/a/346458
So the rendering of the affected chars would be identical, but the underlying byte sequence is not. My guess would be that Rockbox simply checks the byte sequence, and that doesn't match because of the normalization. Not sure what to do about this since those are two versions of the same representation.

Hosting things in the forums is generally a bad idea since things get lost pretty fast. The wiki is a much better place for such thing. We already have https://www.rockbox.org/wiki/UsefulTools so if that's available publicly already (like on github) you could just add a link to that page.

[Pommes]

Thank you, yes your explanation makes completely sense. I should’ve come here and ask before, it took me quite a while to figure the normalization of utf is different on Mac.
Yes, the ? On Mac seems to be different. And to keep from future problems, I just removed all the symbols from file and folder names and from the tags itself, too.
I will ask my friend, if he is willing to upload his m3U8 utf converter to GitHub, I will post a link in the Wiki then.

[Bilgus]

lua has no sense of unicode as long as you could see the file in Demos>lua Scripts>File Browser, (or open it in the File hex viewer) then its quite possible to just do it on device as needed