Rockbox Technical Forums

Support and General Use => Theming and Appearance Customization => Topic started by: bascule on January 15, 2007, 04:52:36 AM

Title: How do I use/select specific characters from a font map?
Post by: bascule on January 15, 2007, 04:52:36 AM
I've been looking at the font maps on rasher's site (specifically nimbus-12 [link] (http://rasher.dk/rockbox/fontstats/Rockbox%20fonts/nimbus-12.png)), but I don't understand how to generate any given character.

For example, I want to to use the Greek Xi (Ξ) character in a WPS, but when I use that character in the WPS file (inserted using Window's 'Character Map' utility), it just displays on the screen of my H120 as the default dotted rectangle character (top left on the font map).

I presume this is because the rockbox fonts are a subset of the full character set.

How can I determine what ?unicode/ASCII? character code relates to a position in the font map and then how could I use Notepad++ to cause that character to display properly in my WPS?
Title: Re: How do I use/select specific characters from a font map?
Post by: pixelma on January 15, 2007, 07:19:21 AM
You would have to save the wps file in utf-8 format.

In Notepad++ there is a setting for doing so... go to menu: "Format" -> "Encode in UTF-8"- it's best to also choose "UTF-8 without BOM" (at the moment). While I did not have any problems with these ByteOrderMarks (a specific "sign" at the beginning that defines the text as being utf-8) in wps files, it was found that having it in fmr-files keeps rockbox from reading the first entry (the same line where the BOM is at).

To save it as utf-8 will also make tags that includes these characters work properly in the wps with the right font.   :)

Edit: As for determing what fonts include which character set you can have a look here... http://www.rockbox.org/twiki/bin/view/Main/UnicodeFonts (http://www.rockbox.org/twiki/bin/view/Main/UnicodeFonts). Other than that I do not now where else you can look but I just tried your "Xi" example with a wps using nimbus-12 and it works when saved correctly...
Title: Re: How do I use/select specific characters from a font map?
Post by: bascule on January 15, 2007, 08:13:42 AM
...it was found that having it in fmr-files keeps rockbox from reading the first entry (the same line where the BOM is at).

I had the same problem with custom tagnavi files (and therefore my reason for moving to notepad++), so I'm aware of the BOM issue

You would have to save the wps file in utf-8 format.

OK, did that, still no joy :(

...but I just tried your "Xi" example with a wps using nimbus-12 and it works when saved correctly...

Any chance of posting it, because of course this is driving me nuts, now. I've tried it in the sim and on my H120 and I still get the default dotted rectangle character displayed.

I've also changed the sim and player's default codepage to unifont, still no luck.


EDIT: Well, it certainly does not for me.

Thanks for the Wiki link. I have now changed my player codepage back to Latin-1 and tested all the fonts listed on that page.

With my properly-encoded .wps file, I can now display the Xi symbol correctly in all fonts listed except nimbus-* and ter-*   ???

I'm confused as to what is going on here, but at least some progress is being made. The problem is back to 'which ASCII/unicode character maps to Xi in nimbus'?

This whole unicode/codepage/character set issue has done my head in for so long and despite reading the rockbox Wiki, Wikipedia and other internet articles, I'm almost as much in the dark as I was before  >:(
Anyone got a link to a simple explanation?
Title: Re: How do I use/select specific characters from a font map?
Post by: pixelma on January 15, 2007, 10:39:16 AM
I first prepared a lengthy post when it crossed my mind that there was something about nimbus and greek and I found it in the "(cvs) changes since August"...

Code: [Select]
1 Sep 13:50 | Marcoen Hirschberg | fonts/nimbus-14.bdf 1.8 | some minor changes and add support for greek
1 Sep 13:47 | Marcoen Hirschberg | fonts/nimbus-12.bdf 1.9 | some minor changes and add support for greek

Together with your reference to rasher's page it leads me to the question - what nimbus-*.fnt are you using? The one from rasher's page or maybe you have an older file in your font-folder? (Please ignore the stupid question in case it was just to explain ;) ) I simply loaded the one that was in my fonts-folder (unpatched and self-compiled simulator of 23rd of December)
Title: Re: How do I use/select specific characters from a font map?
Post by: bascule on January 15, 2007, 04:09:31 PM
pixelma, you're a star!

Your theory was correct, I had some 2006 fonts on my player. Downloaded the latest set and all is now OK. Thank you.

So, just to clear it up once and for all:

Any given unicode-compatible font is just a small selection of non-contiguous unicode characters (glyphs), which can be displayed by using the matching unicode glyphs in the WPS file.

Is that correct?

And that leaves one outstanding question for me: What is the codepage setting in rockbox for?

The manual says
Default Codepage:
A codepage describes the way extended characters that aren’t available within the ASCII character set are encoded. ID3v1 tags don’t have a codepage encoding contained so Rockbox needs to know what encoding has been used when generating these tags. This should be “ISO-8859-1” but to support languages outside Western Europe most applications use the setting of your operating system instead. If your operating system uses a different codepage and you’re getting garbled extended characters you should adjust this settings. In most cases sticking to “ISO-8859-1” would be sufficient.


Am I correct in assuming that if using, as in our case, correctly coded text, it makes no difference. The codepage only comes into play if the encoding is unknown?
Title: Re: How do I use/select specific characters from a font map?
Post by: soap on January 15, 2007, 07:25:18 PM
Let me add further to bascule's question, for I was confused also...
Codepage only matters on ID3v1 tags?  If I have proper v2 tags for everything codepage doesn't affect me?
Title: Re: How do I use/select specific characters from a font map?
Post by: pixelma on January 16, 2007, 01:18:15 PM
Glad I could help!  :D

However, I'm by no means an expert on unicode - it's just that I stumbled across the same problem with the umlauts I had in one of my first wps's. Someone in IRC gave me the hint to save the wps-file as utf-8 and it worked.

At the moment I cannot answer the questions that arose but I've become curious to understand the behinds myself now :) . So I'll try to find out some more and will report it back (as I think it can be helpful to others). Or someone with greater insight could chime in...
Title: Re: How do I use/select specific characters from a font map?
Post by: bascule on January 16, 2007, 02:57:03 PM
Now I'm home, I've had more time to investigate this...

Any given unicode-compatible font is just a small selection of non-contiguous unicode characters (glyphs), which can be displayed by using the matching unicode glyphs in the WPS file.

Is that correct?

Yes. And that is true not only of rockbox fonts, but Windows fonts as well. I spent some time looking through a lot of fonts on a Win2000 machine using the Character Map and they almost all contain some of the 'extended' glyphs (such as boxes and mathematical symbols), but none contained all of them.

The other thing that confused me was that on Rasher's page (http://rasher.dk/rockbox/fontstats/) nimbus-12 is flagged as ISO10646-1, which put me off the unicode scent, but it appears that ISO10646-1 is a sort of unicode-lite, employing all the glyphs, but not with some of the flashy underlying abilities to combine characters or do right-to-left text rendering.

Therefore even a unicode-compatible font may not contain all possible glyphs (although rockbox's unifont does, that's why it is so huge...)

Let me add further to bascule's question, for I was confused also...
Codepage only matters on ID3v1 tags?  If I have proper v2 tags for everything codepage doesn't affect me?

Yes. ID3v2.x can contain specified encodings. ID3v1 tags cannot contain a description of their encoding. They should therefore always be in Latin-1 encoding (ISO/IEC 8859-1), as this is what most applications reading ID3v1 will assume. However rockbox, being smarter than your average application, allows the firmware to assume a different encoding for any unspecified text, such that if you had encoded all your existing MP3s with Bulgarian language ID3v1 tags, changing the codepage would make them all legible again.

I think I've finally got my head round all this now ;D

http://en.wikipedia.org/wiki/ISO-8859
http://en.wikipedia.org/wiki/ID3
http://en.wikipedia.org/wiki/Universal_Character_Set
Title: Re: How do I use/select specific characters from a font map?
Post by: pixelma on January 16, 2007, 06:22:51 PM
Ah... thanks for the info :)

But it still leaves me with a correction I have to make to my first post...

To save it as utf-8 will also make tags that includes these characters work properly in the wps with the right font.

No - that is not true!

In case there is no codepage information given in the tags (i.e. ID3v1 standard), Rockbox will use the one defined in your "default codepage" setting to display the tags (that's what it is for - as the manual describes). Sorry for the misinformation! :-\

---
Just another thing that I found while I was searching for information about ID3v2 specifications: paragraph 3.2 in http://www.id3.org/id3v2-00 (http://www.id3.org/id3v2-00) states about ID3 tags...
"If nothing else is said a string is represented as ISO-8859-1 characters in the range $20 - $FF."
Is it safe to say that there could be problems with characters out of the said ISO-8859-1 range if the tag-program messes things up?
Title: Re: How do I use/select specific characters from a font map?
Post by: bascule on January 17, 2007, 04:10:32 AM
To save it as utf-8 will also make tags that includes these characters work properly in the wps with the right font.

No - that is not true!

I think in essence it is true. If you want to use characters outside of Latin-1, the .wps file will have to be utf-8, otherwise the unicode character codes will not be incorporated within it. Once the .wps is correct, then it relies upon the font having the required characters in order to display them.

The codepage option is a kind of safety net underneath all of that to allow ID3v1 strings without the embedded character encoding information to be handled correctly.

I've just tried re-saving my test WPS fle as ANSI, rather than utf-8 and, as expected, the screen now displays garbage where I had some of the fancy unicode graphical characters.

Just another thing that I found while I was searching for information about ID3v2 specifications: paragraph 3.2 in http://www.id3.org/id3v2-00 (http://www.id3.org/id3v2-00) states about ID3 tags...
"If nothing else is said a string is represented as ISO-8859-1 characters in the range $20 - $FF."
Is it safe to say that there could be problems with characters out of the said ISO-8859-1 range if the tag-program messes things up?

Yes. I think generally that most applications then default to displaying the 'unknown character' symbol, either a rectangle or a question mark.
Title: Re: How do I use/select specific characters from a font map?
Post by: pixelma on January 17, 2007, 07:23:38 AM
The codepage option is a kind of safety net underneath all of that to allow ID3v1 strings without the embedded character encoding information to be handled correctly.
This is what I wanted to point out in the former post. Because I think it is important to know that...

...there is no difference in displaying unknown characters in ID3v1 tags whether the .wps file was saved in UTF-8 or ANSI.

To be really sure I did some experimenting with a utf-8- and ansi-version of my WPS, songs with an umlaut in the album tag (ID3v1 tags only) and different default codepage settings.


I thought the case was closed with it but now I am confused (and it honestly wasn't my goal to keep this thread alive unnecessarily  :-\ )...
I retagged some of the songs as ID3v2.3 using "The GodFather" and "Multi ID3 Tageditor" and got the same results as described above. So either the taggers don't write them correctly or Rockbox doesn't use the codepage information from the tags. Does someone know or have tried too?
Title: Re: How do I use/select specific characters from a font map?
Post by: bascule on January 17, 2007, 08:40:58 AM
I retagged some of the songs as ID3v2.3 using "The GodFather" and "Multi ID3 Tageditor" and got the same results as described above.

Did  you change [menu]>General Settings>Playback>ID3 Tag Priority?

Rockbox may have still been looking at the v1 tags... ;)
Title: Re: How do I use/select specific characters from a font map?
Post by: pixelma on January 17, 2007, 10:05:55 AM
Did  you change [menu]>General Settings>Playback>ID3 Tag Priority?
Nice tip, thanks. Actually I hadn't checked it before but I always have "V2 then V1" set.

So I set it again and to eliminate that this setting has an influence I even deleted the ID3v1 tags...
Still no change.
Title: Re: How do I use/select specific characters from a font map?
Post by: bascule on January 17, 2007, 10:14:08 AM
OK...... so that seems to discount that possibility...

the tags showed the "box" with the wrong default codepage - they were correct with the right codepage setting.

So where/how are you viewing the tag information in rockbox i.e., ID3 viewer, database view etc.?

Reading through some of the old posts concerning unicode and the conversion of rockbox to unicode/utf-8 late in 2005, it seems that there were some problems with some screens not working correctly following the change. Maybe there are still some unconverted parts of the code?
Title: Re: How do I use/select specific characters from a font map?
Post by: pixelma on January 17, 2007, 12:07:41 PM
Since we were talking wps... it was in the wps (but Database and ID3 viewer too) though that's not so important anymore because I found the reason...

Is it safe to say that there could be problems with characters out of the said ISO-8859-1 range if the tag-program messes things up?

Yes. I think generally that most applications then default to displaying the 'unknown character' symbol, either a rectangle or a question mark.
Definitely!

This is what happened to me with the two above mentioned tag programs (maybe there is an option for it in either one but I couldn't find it yet). Before replying I asked around and someone recommended "mp3tag" to me and ID3v2.3 written by this program work fine :). The related setting is also easy to find but it defaults to UTF-16 anyways.

Hope all is clarified now! :D
Title: Re: How do I use/select specific characters from a font map?
Post by: bascule on January 18, 2007, 03:52:33 AM
Good; I'm glad you got it sorted. MP3Tag has always worked really well for me.

I'm just a bit surprised that TGF did not work, as I've always heard good things about it (but I've never used it).