Some characters are shown incorrectly on the playing menu

[jsr_278]

That happens with mp3 files which id3 tag contains characters like ñ, Ñ, á, Á, é, É, í, Í...  These characters are shown with a square symbol on the playing menu. It happens since this commit: https://git.rockbox.org/cgit/rockbox.git/commit/?id=e334a1f95e8149adefbcf1aeccd0be2649106c6a

[iPodVT]

That happens with mp3 files which id3 tag contains characters like ñ, Ñ, á, Á, é, É, í, Í...  These characters are shown with a square symbol on the playing menu. It happens since this commit: https://git.rockbox.org/cgit/rockbox.git/commit/?id=e334a1f95e8149adefbcf1aeccd0be2649106c6a

I started noticing this recently, too, and coincidentally I was just now getting around to investigating it on an ipod6g running Rockbox 4.0 when your post popped up.  I hadn't noticed that my problem tracks are .mp3 files, but now I see that they are.  It's very strange.

In the Database Browser I'm seeing the 'é' in "Béla Fleck" show up as a dotted rectangular symbol under Artist, but under Track I'm seeing the track name - "Whitewater - Béla Fleck" correctly displayed with the 'é' character.  As well, in the WPS I'm seeing the same discrepancy between Artist and Track.  So I went back into the source of my iPod's music (which is my master iTunes Music Library) and I copied and pasted "Béla" from the Track tag to the Artist tag, resynced my iPod, rebuilt the database, and rebooted the iPod, but I'm still seeing the same discrepancy.

There is at least one other similar example in my music library: Artist "Dédé".  With that one, when I have the font set to 14-Rockbox-Mix instead of the default 15-Adobe-Helvetica, I'm seeing the first 'é' displayed as a circle with a question mark inside it and the second 'é' displayed as a dotted rectangle.

I loaded those two tracks onto an ipodvideo running dev build c71b6265b0-240917, and those Artist names are both displaying correctly in both the Database Browser and the WPS.

Then I created a .txt file on my Mac, named it "Béla Dédé.txt" with "Béla Dédé" as its contents and copied the file to my same ipod6g running RB v4.0.  The filename was correctly displayed in the File Browser and the contents were correctly displayed in the Text Viewer plugin with Text Viewer's encoding set to UTF-8, but when I switched encoding to ISO-8859-1 (which I believe used to be the default, IIRC) the display showed "é" for each of the 'é' chars.  I looked for a way to change the Encoding setting for Rockbox as a whole but didn't find it (maybe it's determined by the Font setting?).

That's as far as I've gotten with this so far.

[iPodVT]

I'm also seeing the 'é' char incorrectly displayed in the Database Browser under Composer for those same .mp3 tracks.

[speachy]

That happens with mp3 files which id3 tag contains characters like ñ, Ñ, á, Á, é, É, í, Í...  These characters are shown with a square symbol on the playing menu. It happens since this commit: https://git.rockbox.org/cgit/rockbox.git/commit/?id=e334a1f95e8149adefbcf1aeccd0be2649106c6a

The metadata in these files is probably Latin-1 (or Windows-CP1252) instead of UTF-8.

That's independent of the *filename* which (given the fat32 filesystem) is all but guaranteed to be UCS-2 or UTF-16 if it was copied from an operating system released in the past twenty years or so...

There's a setting in rockbox to change the default codepage but you *really* should fix the metadata instead.

  https://download.rockbox.org/daily/manual/rockbox-ipod6g/rockbox-buildch8.html#x15-1530008.5

[iPodVT]

The metadata in these files is probably Latin-1 (or Windows-CP1252) instead of UTF-8.

That's independent of the *filename* which (given the fat32 filesystem) is all but guaranteed to be UCS-2 or UTF-16 if it was copied from an operating system released in the past twenty years or so...

There's a setting in rockbox to change the default codepage but you *really* should fix the metadata instead.

  https://download.rockbox.org/daily/manual/rockbox-ipod6g/rockbox-buildch8.html#x15-1530008.5

Thanks very much for your reply.

I retyped one of the offending tags in iTunes, resynced my iPod, and now it's displaying properly in Rockbox.  But in order to make the edit 'stick' in iTunes I couldn't just retype what looked to be the same thing as what was already there - I had to make some other additional minor change, close the editing window, reopen it, and then undo that additional minor change.  iTunes was trying to be too 'smart'.

If you are able to explain in a brief sentence or less, please tell us what changed in Rockbox that made these tags start displaying differently?

And I don't understand why the first 'é' in "Dédé" would display as a char that looked like a question mark in a circle while the second 'é' would simultaneously display as a dotted rectangle char, when the font is set to 14-Rockbox-Mix?

[bahus]

To return old behavior change Settings -> General Settings -> Display -> Default Codepage to Latin1 (default was changed to UTF-8)

[jsr_278]

Thank you very much for the information, I didn't know about the codepage encoding. I tried both solutions and they work absolutely fine.

[speachy]

If you are able to explain in a brief sentence or less, please tell us what changed in Rockbox that made these tags start displaying differently?

And I don't understand why the first 'é' in "Dédé" would display as a char that looked like a question mark in a circle while the second 'é' would simultaneously display as a dotted rectangle char, when the font is set to 14-Rockbox-Mix?

https://en.wikipedia.org/wiki/Character_encoding has a very comprehensive explanation of how this works.

But the shorter answer:  The byte sequences in the metadata are presumably Latin1, for which each byte equals one "letter" but limits you to a max of ~220 possible individual letters.  This is completely inadequate for representing the *1.1 million* (or so) characters currently defined, which is why modern encoding systems use multiple bytes to encode each letter, allowing multiple lanuguages to be displayed simultaneously.  UTF-8 in particular uses _variable_ numbers of bytes per letter in an attempt to be more efficient wile still maintaining 1:1 compatibility with the OG encoding, ASCII (which only defines ~90 letters)

So while the byte sequences in your metadata are valid under Latin-1, they represent _invalid_ UTF8 sequences, which causes the decoder to hiccup and produce garbage before it re-synchronizes at the next valid sequence.  As those two 'é' have different bytes preceding them, they decode differently.

[iPodVT]

Thanks very much, speachy, for all of the information.

And thanks, jsr_278, for initiating this topic.