Different encodings in Affinity & MS Word !

[Yves Michel]

Some unicode characters used in Latin languages are ignored by Affinity Publisher. And worse, they are replaced in a text zone by other characters.

As I'm using Affinity to publish my fonts, this is a very bad problem !

[Yves Michel]

The missing Unicode characters: 0100 0101 0116 0117 0122 0123 012A 012B 012E 012F 0136 0137 013B 013C

[Alfred]

Welcome to the Serif Affinity Forums, @Yves Michel.

Apologies if this is a silly question, but does your chosen font include glyphs for all of the characters in the Latin Extended-A Unicode block?

[Yves Michel]

The illustrating picture is in Arial Regular. Difficult to have something more universal.

Like I wrote, this is a template to show my fonts, so, of course, the fonts tested here include these glyphs.

There are no silly questions! Sometimes silly answers! ☺️

[kenmcd]

There are no "Ignored Unicode characters".
See those characters in the image below - Arial, Aptos, Inter, Noto Serif.

Inter is used in this forum: ĀāĖėĢģĪīĮįĶķĻļ

My guess is you have some junk font installed which is based on Arial and it is causing this issue - because it still has "Arial" in some of the name fields.

[Yves Michel]

Using the same Word document as a start, I have the exact same result using Calibri, Tahoma, or any other fonts. Difficult to name them "junk fonts".

To be complete, I tried copy/pasting from Word to other software: 

• From Word to WordPad, some missing glyphs but not the same as in Affinity.
• From Word to Notepad +, Everything OK!

And from the "Character table" to Affinity, no problem!

If you have an explanation, as an Affinity beginner, I would be happy to learn.

[Old Bruce]

22 minutes ago, Yves Michel said:

Using the same Word document as a start, I have the exact same result using Calibri, Tahoma, or any other fonts. Difficult to name them "junk fonts".

To be complete, I tried copy/pasting from Word to other software: 

• From Word to WordPad, some missing glyphs but not the same as in Affinity.
• From Word to Notepad +, Everything OK!

And from the "Character table" to Affinity, no problem!

If you have an explanation, as an Affinity beginner, I would be happy to learn.

I have learned that if @kenmcd makes a suggestion about fonts the suggestion is most likely right. So, how many fonts do you have installed? Do you have any way of finding out if you have more than one set of fonts named "Arial". For what it is worth here on my Mac with the bog standard Arial I see the correct glyphs being used in Publisher. kenmcd uses Windows.

The fact that going from Word to WordPad gives you some missing glyphs most likely means there is a problem on your machine.

[Yves Michel]

Thank you all.

But this has nothing to do with the fonts used. The ones I used are all original Windows fonts, no Junk ones!

I don't have more than one set of Arial. And the fonts I create and test have all the needed glyphs in my example.

I repeat there is no problem when I paste in Notepad +. And no problem again in Libre Office Writer, Fontlab 8, Paint, Inkscape, Krita, Corel Paint Shop 2023, etc. So, no problem with the Windows clipboard.

So, excuse me if I suspected a problem in Affinity. And It's not very pleasant to hear that you use junk fonts or that your machine has problems. This remember me of some software vendors who rejected all bugs on Microsoft.

 

[Evaluation complete.]

Could you share the document at the top so I can view it on my machine?

I have no idea what might be influencing this, but perhaps you could briefly describe your machine (operating system, version) and the installed keyboard language/your language setup on the operating system.

And yes, a sample Word document with the same characters.

[kenmcd]

2 hours ago, Yves Michel said:

Using the same Word document as a start,

This could be part of your problem.
What font was selected in Word when you copied the text?
Was it your font?
The characters may have already been changed when copied.

Was the Word doc an old document?
Which could have been made with old fonts and old encoding.
That could explain the different behavior in different applications.
Some applications understand the old encoding (Affinity does not) and convert it to Unicode.

The odd characters make it look like it could be an encoding issue.
But that usually affects all the characters.

Can you provide the Word test doc?
And the font(s) from that doc if needed.

Nobody else has had this problem, so it is definitely something at your end.

[kenmcd]

Could also be an issue with how you entered those characters.
As a unique character code-point, or as a base character plus a combining diacritic.

We need to see that Word doc.

[kenmcd]

The plot thickens...

I pasted this into a new Word doc: ĀāĖėĢģĪīĮįĶķĻļ  (which is Unicode)
Looks fine.

Then copied from Word and pasted that into APub - and got the garbage.

So I then tried to paste from Word to UltraEdit - and it warns me the text is not Unicode - it is Latin 1252 ANSI - the old pre-Unicode code page encoding.
So Word is sending old Latin 1252 ANSI encoding to the clipboard with those characters.
Which is bizarre (that is Apple sort of nonsense).

Some applications understand the old encoding so they display the text fine.

Pasted this into LibreOffice: ĀāĖėĢģĪīĮįĶķĻļ
Then copied that from LibreOffice to APub - looks fine.

Work-around - don't use Word for your sample text.

[Evaluation complete.]

Yes, I can confirm that. Also on Mac, a roundtrip through Word corrupts these characters, while they are preserved if you use Pages as an intermediary before Publisher. That was useful information for the project I’m working on, so thank you for the clarification.

[Yves Michel]

Thanks all for racking your brains on this problem!

And thanks to kenmcd for finding the problem and offering a solution! I never used (or even knew of) UltraEdit but it shows the errors!

So, in the future, I will start from Libre Office or Notepad++ (never from Word) to copy/paste a text in Affinity.

And, like the software vendors I mentioned, blame it on Microsoft! 😂

[Yves Michel]

Furthermore, Notepad++ includes an Encoding menu where it's possible (among other features) to convert old ANSI encoding to modern UTF-8 (and vice versa)

One could ask why Affinity doesn't use the same detection of encoding method as Libre Office or Notepad++. Maybe because it's not turned to the past but to the future?

[kenmcd]

This was bugging me... those characters (ĀāĖėĢģĪīĮįĶķĻļ) are not in the Latin 1252 ANSI code page.
And maybe that is why they did not work.
I did not test your other characters, but if they ARE working - that means APub is now converting old encoding.
Awhile back I did notice the cmap files used to translate the encodings are in APub.
On Windows the files are here:
C:\Program Files\Affinity\Publisher 2\Resources\Standard CMaps\
And String Encodings here (including 1252)
C:\Program Files\Affinity\Publisher 2\Resources\String Encodings\
At the time I thought that was curious since APub did not support the conversion.

That appears to have changed.

So to test I got most of the 1252 and did the copy and paste from Word to APub.
It worked.
The problem with ĀāĖėĢģĪīĮįĶķĻļ is they are not in 1252.
So Word identifying them as that is a problem.

The characters I pasted into Word and then into APub are below.

Windows-1252
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~€‚ƒ„…†‡ˆ‰Š‹ŒŽ‘ʼ“ˮ•–—˜™š›œžŸ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúû

I do not know if ĀāĖėĢģĪīĮįĶķĻļ are in any code pages.
Have to check that and do some more testing - to see if the old encoding is properly identified (as it is not 1252) that it then works.

But the big take-away is...
Conversion of the Windows-1252 code page to Unicode is now working.

Hmmm... have to test the old MacRoman encoding to see if it is working too.

Those are the two most common old encodings.
Windows-1252, and on Mac the MacRoman.

Here in the forum we still see MacRoman encoding in PDFs from ID and other macOS apps. Which has in the past been a problem when opening those PDFs in APub.
Guess that will not be a big a problem now.

[Yves Michel]

Thank you Ken for all your research on that matter!

An expert typographer of my friends recommanded Affinity to me lately . Before that, I wasted a bit of time on Paint Shop Pro. This explains I have still to master this wonderful Affinity suite.

He also pointed 2 other ways to solve the transfer problem:

• a useful Affinity utility. In French, it's in "Fenêtre > Texte > Navigateur de glyphes". This should be something like "Window > Text > Glyphs navigator". From there, it's possible to copy glyphs from installed fonts to a text box by double-clicking them
• we use Fontlab 8 and, in FontLab’s Font Window, I can select glyphs and use “Text > Copy text as > Unicode” to copy a Unicode string with the characters of the selected glyphs to the clipboard. Then paste into an Affinity document.

Thanks again!

[kenmcd]

10 minutes ago, Yves Michel said:

we use Fontlab 8 and, in FontLab’s Font Window, I can select glyphs and use “Text > Copy text as > Unicode” to copy a Unicode string with the characters of the selected glyphs to the clipboard. Then paste into an Affinity document.

That's how I got the Windows-1252 characters above.