Publisher text editor does not recognize cyrillic text

[anto]

Version 2 (from 2.0 to 2.3.0.2157) of Publisher does not work properly with Cyrillic text. They often fail to recognize letters and split them into their component parts.
I can't reproduce this when I want to. The text is usually copied and pasted from office documents - libreoffice or word. This happens with two or three words in the text. Today it happened with 7.

MacOs Ventura

 

 

 

[walt.farrell]

Can you share the .odt or .docx document you're copying from?

[anto]

20 minutes ago, walt.farrell said:

Can you share the .odt or .docx document you're copying from?

  Here is file and video

 

 

2_5420552630847946851.doc

[kenmcd]

Looks like the diacritics are either being decomposed, or they were entered separately in the original document.

If they are entered separately usually the shaper will/should turn these into the single character. 

What is the original font? 

On my phone at the moment, but can look at this on my laptop in about an hour.

Note: In your movie the single dot reappears on the decomposed character which could be a font issue, or an issue with the shaper.

[anto]

32 minutes ago, kenmcd said:

Looks like the diacritics are either being decomposed, or they were entered separately in the original document.

If they are entered separately usually the shaper will/should turn these into the single character. 

What is the original font? 

On my phone at the moment, but can look at this on my laptop in about an hour.

Note: In your movie the single dot reappears on the decomposed character which could be a font issue, or an issue with the shaper.

It doesn't depend on the font. The letters were entered in the normal mode. This does not happen with all words. There are many other words in the text, but this happens only with some of them, there are the same words, but they are spelled correctly.

 

 

[kenmcd]

A font without any composites would affect this - and probably work.
So I doubt it always "does not depend on the font."

This appears to be a problem with the Word shaper, the font composites, and the Affinity shaper.
When you say it was "entered in the normal mode" do you mean in Word?

In Liberation Serif the Ukrainian yi-cyrl (ї) U+0457 is composed of two glyphs.
A dotlessi (ı) U+0131 and a diresis (¨) U+00A8.
That is the old legacy diresis, not the diresiscomb  (   ̈ ) U+0308 which it should be.

Word apparently sees this and changes the diresis to the diresiscomb U+0308
but it also changes the dotlessi to a ibyelorussianukrainian-cyrl (і) U+0456.
Then apparently the Word shaper removes the dot and applies the diresiscomb.
Because of that it appears to be the yi-cyrl  (ї) U+0457.
But when the text is copied - that is what comes over - those two characters.
ibyelorussianukrainian-cyrl (і) U+0456 and diresiscomb U+0308
And that is what you are seeing in APub.

In APub when I directly enter the yi-cyrl (ї) U+0457 as Unicode - it works fine.
If that correct character was coming over from Word - it would work fine in APub.

I am not sure if the Affinity shaper should be "fixing" this mess.
That would basically be fixing a Word-created error.
But that could be another help-the-users-deal-with-this situation.

Word compounded the error when it replaced the dotlessi.
If it just replaced the dieresis with dieresiscomb - that should work.
Or replace the whole thing with yi-cyrl (ї) U+0457.
(which is what I think harfbuzz does).

[walt.farrell]

I wonder if it would work better to Open a .docx file rather than using copy/paste. I can try that when I'm at a different machine, but I'm not sure I'll recognize whether the problem has occurred  

[kenmcd]

Yes, I wanted to take a look inside the document.
But because it was .doc, and not .docx - cannot look inside.
Like to see what actual characters are in there.

There could also be an issue with the .doc.
When I opened it in LibreOffice Writer the font is shown as:
Liberation Serif;Times New Roman

Which of course makes no sense.

Word just showed: Liberation Serif

[MikeTO]

I already tried it with docx and rtf but there was no difference.

test.docx

[kenmcd]

Note to self: you can save a .doc file to a .docx... and then look inside. Duh.

The characters for yi-cyrl (ї) in the .docx are what is coming over to APub.
ibyelorussianukrainian-cyrl (і) U+0456 and diresiscomb U+0308

Kinda odd.

 

 

[anto]

The same problem in Windows.

And when i use File-->Place

[walt.farrell]

14 hours ago, kenmcd said:

The characters for yi-cyrl (ї) in the .docx are what is coming over to APub.
ibyelorussianukrainian-cyrl (і) U+0456 and diresiscomb U+0308

I'm not quite sure I understand what you mean there, but thanks and congratulations for the detailed explanations and detective work in this topic!

Are you saying that the proper character was contained in the Word file, composed correctly, but the Affinity application broke it during import?

[kenmcd]

2 hours ago, walt.farrell said:

Are you saying that the proper character was contained in the Word file, composed correctly, but the Affinity application broke it during import?

No. Word broke it. And Affinity is just displaying what is there.

This could be something which could be fixed during the Place processing of the Word file. A new test/fix to look-out-for-dumb-stuff-like-this and replace it with the one correct character. May be difficult to actually implement, but would help users and prevent issues like this in the future.

Part of the problem may be in how the font is constructed. Current best practices are to only use combining accents in composite characters, not legacy accents. This could be Word's (DirectWrite's) way of dealing with older fonts which still used legacy accents in composites and did not have a dotless i glyph. Dunno. I can only guess.

If we de-compose the font (make all the composites go away), it would probably not be an issue.

@anto If I make you a decomposed version of that font, would you test it for us? (would probably not be until tomorrow) 

[kenmcd]

This could be a keyboard entry issue.

When I enter the yi-cyrl (ї) into Word using the Unicode code (0457 + Alt-X), and then paste that into APub it works fine.

@anto What keyboard are you using?

 

[anto]

6 minutes ago, kenmcd said:

What keyboard are you using?

qwerty with numeric

6 minutes ago, kenmcd said:

This could be a keyboard entry issue.

But the problem is that the file was sent to me.

[anto]

I checked this text in Indesign. It behaves the same way.

[kenmcd]

21 minutes ago, anto said:

But the problem is that the file was sent to me.

That is a major problem.

Above you said:

22 hours ago, anto said:

The letters were entered in the normal mode.

Which to me means you typed it on your keyboard into Word. 

But instead, you apparently received a defective file. So the source of those odd characters is completely unknown. 

This has been a giant waste of time.

[anto]

9 minutes ago, kenmcd said:

Which to me means you typed it on your keyboard into Word. 

But instead, you apparently received a defective file. So the source of those odd characters is completely unknown. 

This has been a giant waste of time.

This is not the first time this has happened. It happens very often. I get files from different people. Everyone can't type the text wrong. Everyone has different versions of programs (Word, for example, LibreOffice etc). Even in this text, there are the same words, but spelled correctly and not decomposed into elements.

[Hangman]

FWIW 1
According to good old Google "The Liberation Fonts (Sans and Serif) are a font family which aims at metric compatibility with Arial, Times New Roman and Courier New sponsored by Red Hat" so the Liberation Serif;Times New Roman perhaps makes a little sense, not from a font selection perspective but just in general and as far as I know it is the default font used in LibreOffice.

FWIW 2
If I change my keyboard to either 'Ukranian - QWERTY' or 'Ukranian - Legacy' on macOS and type the words in Microsoft Word then both 'copying and pasting' the text from Word or 'placing' the Word file in Publisher displays the text correctly, i.e.,

Instead of a ibyelorussianukrainian-cyrl (і) U+0456 and diresiscomb U+0308 you have the U+0457 Unicode character ї and
Instead of a ibyelorussianukrainian-cyrl (и) U+0438 and diresiscomb U+0306 you have the U+0439 Unicode character й

This to me suggests the issue is one of text input and not an issue with Publisher...

Sample Word File

Source.docx

[anto]

1 hour ago, kenmcd said:

Which to me means you typed it on your keyboard into Word. 

But instead, you apparently received a defective file. So the source of those odd characters is completely unknown. 

This has been a giant waste of time.

The text was typed in LibreOffice on Linux Mint

[kenmcd]

8 hours ago, anto said:

The text was typed in LibreOffice on Linux Mint

"was typed" LOL - by whom? The Invisible Man?

This appears to be exactly opposite from my initial understanding/assumptions.

I assumed you, or The Invisible Man, had entered the characters correctly,
and the shaper had broken it.

But it appears that The Invisible Man (TIM) had entered the wrong characters,
and the shapers actually hacked the display to show the intended character.

The yi-cyrl (ї) U+0457 character is in Ukrainian, and not in Russian.

My guess is that TIM is using an old Russian flavored keyboard layout, and has to use some old trick to enter the character.
And the shapers in Word and LibreOffice are aware of this old hack, and they display the correct glyph - but the underlying characters are still wrong.

Unfortunately this has the effect of standardizing the hack.
When the correct fix is to correct the character input.

Windows has three different Russian keyboard layouts available, and also has two Ukrainian keyboard layouts available. And there are apparently some popular 3rd-party Russian keyboard layouts.
I assume there are similar keyboard layout choices available for Linux.

The best fix is for TIM to use a keyboard layout or tool which enables him to enter the correct character codes, not some hack which the shaper has to hack visually.
@Hangman demonstrated one working solution above.

Or you could use find-and-replace to correct the characters.

I hope Affinity never adds this hack to their shaper as it just perpetuates this.
Affinity is just correctly showing what is actually there.
What is there should be corrected by the author.