Non-Roman Scripts

[tslarkin]

I was making a book for my neighbors' 50th anniversary. I came up with the idea of constructing a title page with a background showing "fiftieth" in as many languages as I could find. Of course there's a web site for this. I selected text from the page, did some RegEx processing, and pasted the result into a BBEdit window with UTF-8 encoding. (BBEdit is a plain text editor.) I attach this file, which contains lines like this:

canfed, 50, հիսուներորդ, əllinci, পঁচাশত্তম, 第五十, 第五十,
ორმოცდამეათე, પચાસમું, पचास, fiftieth, 五十, ಹದಿನೈದನೇ, елуінші, ហាសិប,
50 분의 1, ຫ້າສິບ, അമ്പതാം, पन्नासाव्या, тавь дахь, ငါးဆယ်မြောက်သော,
पचासवां, පණස්, fiftieth, ஐம்பதாவது, పదిహేనవ, ที่ห้าสิบ, ellinci,
پچاسویں, elliginchi, Năm mươi, خمسون, חֲמִישִׁים, پنجاهمین, vyftigste,

As you can see, the various scripts display correctly in a single font. I don't know what font this post uses, but the BBEdit file shows the scripts using Courier New.

Fine, i thought, I'll just paste this into Publisher, which produced this: (I include this as a graphic, because if I copy and paste from Publisher into this post, I get the correct glyphs above, and not what I see in Publisher, which suggests that there is nothing wrong with the UTF-8 data.)

As you can see, many glyphs appear as empty boxes, others are incorrect, and others are clearly from other fonts.

I tried several other experiments involving InDesign, Pages and .docx files. None of them, with their powerful font handling, could accomplish what a plain text editor or an HTML page could do, namely to show all the glyphs in their correct form without font changes and strange variations in line height. 

I don't understand this. Is Publisher not UTF-8 compliant? What am I missing? Why does Safari or a plain text editor do a better job of representing non-Roman scripts?

fiftieth.txt

[walt.farrell]

Welcome to the Serif Affinity forums.

Publisher does support UTF-8, but UTF-8 does not indicate what font should be used. Publisher expects you to indicate what font should be used, and I believe it expects the font you choose to have all the codepoints you want to use.

My guess would be that the other program you're using is performing font substitution and locating an installed font that supports each codepoint. Publisher does not do that, as far as I know. If you choose to use font X and X doesn't support some codepoint then you'll get those empty rectangles.

 

[tslarkin]

I specified Courier New in Publisher, which is the exact same font that represented the text correctly in BBEdit and in Safari.

[walt.farrell]

12 minutes ago, tslarkin said:

I specified Courier New in Publisher, which is the exact same font that represented the text correctly in BBEdit and in Safari.

Browsers definitely perform font substitution. I'm not sure about BBEdit, but my guess is that it does, too.

Publisher does not.

[Old Bruce]

When I opened the text file with BBEdit I got this

I think you would be better served by using the website you mentioned and copying from that and pasting into Publisher or Designer going line by line for each language.

[tslarkin]

Old Bruce, open the file in BBEdit, then try Reopen Using Encoding UTF=16.

[Old Bruce]

2 minutes ago, tslarkin said:

Old Bruce, open the file in BBEdit, then try Reopen Using Encoding UTF=16.

Bit thick here this morning. Works. Thanks.

[tslarkin]

Walt,

You are correct. There's some font substitution going on in the background.

Publisher does do automatic font substitution in a way that's perplexing. If you paste հիսուներորդ into a text box, you will see the correct glyphs. However, if you select the text, the font selector in the upper left will indicate that it is "! <whatever font you choose>". If you activate the drop down font menu and select different fonts, the glyphs will not change (you can see this clearly if you extend the selection to some Roman text), unless you select a font like Noto Sans Armenian, which does have the glyph.

In other cases, Publisher shows a box if the font does not have the glyph for the character. If you paste पन्नासाव्या into a text box, you get empty boxes unless you select a font that supports Devanagari.

[tslarkin]

I have concluded that no single font contains all the glyphs necessary to display the text. If you encounter a similar problem, I recommend beginning with a font that's very inclusive, such as Code2000 or Arial Unicode. You may see some empty boxes. In addition, if you look carefully, (and this is difficult because the problems will occur in languages and scripts you don't know), you may find other, less obvious errors. Even if you don't know a particular SE Asian script, for instance, the presence of a Roman character with a diacritical mark within that script suggests something is amiss. Here, the Mac program UnicodeChecker (earthlingsoft.net) comes in handy. Select the suspect character or rectangle, paste it into the box in UnicodeChecker, and the font selection menu above the box will show all the installed fonts that contain a glyph for the character. Then return to Publisher and change the font for the word or words incorrectly displayed.

Probably there's an equivalent of UnicodeChecker on other platforms.

Thanks for the help.

[walt.farrell]

1 hour ago, tslarkin said:

Probably there's an equivalent of UnicodeChecker on other platforms.

BabelMap offers similar functions on Windows.