Publisher interpreting colons incorrectly in placed PDF

[stutes]

Affinity Publisher v1.7.2

Mac OS X 10.14.5 (18F132)

My workflow:

• In a separate calendar app (BusyCal) I edit calendars that I've imported from Google Calendar.
• I export the calendar to PDF by printing it to PDF.
• I place the PDF into my newsletter.

The results:

• Publisher is incorrectly interpreting the colons in the PDF. Instead of displaying colons, it is displaying an odd looking "A".
• I've attempted this several times, and have even tried using different fonts in my calendar app, thinking that Publisher doesn't like the font. Results are the same each time.
• I am able to edit the "embedded" PDF (it's supposed to be linked) within Publisher, and can replace the "A"s with colons manually, so it does not appear to be a font issue.
• I'm not able to find and replace, as Publisher isn't able to display this "A" in the find field.
• The same PDF file displays correctly in Preview.

Expectation:

• Publisher places/links to the PDF file as is

Calendar LINKED - OVERWRITE letter.pdf

[kenmcd]

Most likely those are not actually colons.
Some fonts have an OpenType feature which makes times look better by using a raised colon.
The contextual alternates feature sees the colon is between two numbers and it replaces the colon with one which sits a little higher.

What font is this?
And is it OpenType PostScript or OpenType TrueType ?
Were any of the fonts you tested OpenType PostScript?

There may be some work arounds.
On my phone on the Metro at the moment so not going to elaborate at the moment.

[stutes]

I was using what Busycal referred to as "System Font Regular" originally, which I think on MacOS High Sierra is San Francisco. It MAY be an OpenType Postscript, but I'm basing that info off of other fonts from Apple; I can't find the font in my font manager. 

The alternate I tried was Walkway Condensed Bold. It is a TrueType font. It does not have any alternates, stylistic sets nor character variants when I open the Typography panel in Publisher.

I'd be interested in workarounds. I suppose I continue trying other fonts, but ones I've tried in the past don't work very well for my layout. The calendar software doesn't offer lots of type adjusting features, aside from picking another font. Most other fonts either have too much character spacing, or line spacing, and I need to economize in all those regards to fit as many lines of text as possible into each day on the calendar.

Exporting a PDF from my calendar software and placing them into my layout has worked for me in other desktop publishing software. Would be nice if Affinity Publisher could do the same. I don't need the characters in the PDF interpreted; I just need the vector info contained within the PDF displayed and printed as is.

[kenmcd]

Very interesting ...
The text font in the PDF is: .SFNStext
The heading font in the PDF is: .SFNSDisplay-Bold
Both are TrueType fonts (which is surprising).
I guess the leading "." means it is a system font on the Mac.
I do not know if SF is "San Francisco" or "System Font" -- not that familiar with Mac fonts.

The San Francisco fonts from the Apple Developer website are OpenType with PostScript outlines.
I checked and those fonts do have an alternate raised colon which is applied by the OpenType Contextual Alternates (calt) feature. The calt feature is On by default so it will automatically replace the colon in between numbers in a time (in these fonts).

APub seems to do better with importing the correct characters when the font is OpenType PostScript.
So that is why I wanted to know which fonts you tested.
It appears you have only tested with TrueType outline fonts.

Attached below are the San Francisco fonts from the Apple developers website.
These fonts are OpenType with PostScript outlines.
The font names are different so you should have no issues with other installed fonts.
Please give them a try.

SF Pro Display and Text.7z

Busycal could also be using specific characters from fonts it knows will be installed in the OS.
Some apps do this with known system symbol and dingbat fonts.

So please try the SF Pro fonts and post the new PDF here so we can see what is happening.

[thomaso]

Possibly the error happens already when generating the pdf in BusyCal as a issue with encoding the text format.

In Acrobat I can save the pdf as text, there I have the option to set the text encoding:

The result is a 74 byte file with just the headline of the table. Not useful.

I also can save the PDF in Acrobat as .RTF. This shows the colons as letter b .
When I copy/paste the content from PDF in Acrobat to a text editor it shows the colon as '' .

[music2bmade]

I am seeing strange font discrepancies in PDF's too, that none of my other publishing programs (Pages and Swift Publisher) are struggling with. Sometimes a word will be on a whole separate line from where it is supposed to be. Publisher is acting like it can't find the font and is doing substitution. PDF's should have the fonts embedded and thus never be lacking for those fonts, true? 

I can't attach any files because of copyright. (My files are music saved as PDF's.)

My workaround has been to use Preview in Mac to save the file as a tIff which Publisher reads perfectly.

Other than this issue I am REALLY enjoying this program. I am doing less and less on my other publishing apps.

 

 

 

[garrettm30]

11 hours ago, music2bmade said:

PDF's should have the fonts embedded and thus never be lacking for those fonts, true? 

This is currently a known limitation of Publisher. It does not yet support font passthrough, so you must have the font installed on your computer, or else there will be substitution.

Since you mention music, I think this also has implications for music as well. Some people have reported that PDFs exported through various music programs get completely destroyed when placed in Publisher. For example, when I try to embed PDF output by Lilypond, my notes just get reduced to headless stems and slurs of nothing:

That might depend on the music program. Finale used to (does it still?) have installed fonts for that purpose. It mucked up the font menus of all other programs with irrelevant fonts, but perhaps it would mean you could place a PDF in Publisher more reliably.

[Alfred]

On 8/27/2019 at 9:53 PM, stutes said:

Instead of displaying colons, it is displaying an odd looking "A".

That “odd looking” letter is A ogonek, used in Polish and a few other languages. 

[stutes]

2 hours ago, garrettm30 said:

This is currently a known limitation of Publisher. It does not yet support font passthrough, so you must have the font installed on your computer, or else there will be substitution. 

This is good to know, but personally this isn't the issue (as the OP) I'm experiencing. I'm using calendar software (BusyCal) on the same Mac I have Publisher installed, so when I create the PDF from BusyCal, it is using local fonts to create it. So if the above were entirely true, Publisher shouldn't have issue bringing this PDF, but it does. I assume it is because Publisher tries to interpret the characters in the PDF, so the PDF can be editable. I'm sure that many appreciate the ability to edit PDFs, but I personally don't need it very often.

I wish Publisher provided the option to turn that off, and just display the PDF as is.

[MikeW]

25 minutes ago, stutes said:

2 hours ago, garrettm30 said:

This is currently a known limitation of Publisher. It does not yet support font passthrough, ...

...So if the above were entirely true, Publisher shouldn't have issue bringing this PDF, but it does. I assume it is because Publisher tries to interpret the characters in the PDF, so the PDF can be editable. ...

In the case of opening a PDF as an editable object, the issue is really whether or not the Affinity applications can utilize the fonts as embedded within the PDF, including the ability to remap improperly encoded glyphs and/or map particular glyphs that are not encoded at all and/or are missing.

Here is the issue with your calendar pdf when a font report is ran:

The colon glyph is present, but the encoding is missing altogether.

Here, though, is how your calendar appears when opened for editing in an application that can utilize embedded fonts:

The colons are present. Further, if I make a new PDF from that application and run a font report on that PDF, the colon is mapped:

The issue, it would appear, is a combination of problems, one of which is likely the calendar application.

[music2bmade]

3 hours ago, garrettm30 said:

That might depend on the music program. Finale used to (does it still?) have installed fonts for that purpose. It mucked up the font menus of all other programs with irrelevant fonts, but perhaps it would mean you could place a PDF in Publisher more reliably.

Thankfully, for all of my Finale work, my clips are saved as tiffs and so are no problem whatsoever in Publisher. It is when I get a PDF from an outside source that the problems arise, probably due to a missing font.

If the ability to read embedded fonts in PDF's is not included in Publisher I would call this a "bug."  It defeats the whole purpose of a PDF which is file sharing and keeping formatting intact. (And every other program BUT Publisher seems to have no issues with PDF's)

[MikeW]

4 minutes ago, music2bmade said:

....If the ability to read embedded fonts in PDF's is not included in Publisher I would call this a "bug."  It defeats the whole purpose of a PDF which is file sharing and keeping formatting intact. (And every other program BUT Publisher seems to have no issues with PDF's)

It's not a bug at all. Just a capability that has not yet been incorporated. No different than other capabilities that have not yet been incorporated.

PDF passthrough will come. Using embedded fonts will come. But they will come later.

[thomaso]

49 minutes ago, MikeW said:

the ability to remap improperly encoded glyphs and/or map particular glyphs that are not encoded at all and/or are missing

On 8/29/2019 at 1:16 PM, thomaso said:

Possibly the error happens already when generating the pdf in BusyCal as a issue with encoding the text format.

In Acrobat I can save the pdf as text, there I have the option to set the text encoding:

It did not work for me with any of those from this list. – Also in a code editor app, where I can switch the encoding, I could not make the colons from that pdf appear. – Can you explain a little more about that, @Mike?

I wonder whether it is possible to save such an affected pdf with a different + fitting encoding, to make its glyphs appear correctly in Affinity.
Also, is there a way to get info about the saved and/or the used/shown encoding of a pdf?

p.s.: what do the  2x 3x 4x  in your screenshot stand for?

[stutes]

I appreciate all the effort that everyone is doing in analyzing my original PDF from BusyCal.

1 hour ago, MikeW said:

The issue, it would appear, is a combination of problems, one of which is likely the calendar application.

So, again, I appreciate the statement, but if my underlying PDF has a combination of problems, I can't figure out why Publisher is the only app that appears to have a problem with it. All other apps read the PDF just fine.

1 hour ago, music2bmade said:

If the ability to read embedded fonts in PDF's is not included in Publisher I would call this a "bug."  It defeats the whole purpose of a PDF which is file sharing and keeping formatting intact. (And every other program BUT Publisher seems to have no issues with PDF's)

With all respect, @MikeW , I'd have to agree with @music2bmade. Publisher's inability to read embedded fonts may not be a bug, but Publisher almost seems to be working hard to defeat the purpose of the file format, just for the sake of being able to edit the file type.

Just now this issue reared it's ugly head in a different way: I downloaded a news bulletin to insert into my handout, and since I did not have the fonts embedded in the PDF, Publisher is not able to interpret the PDF correctly, even though the fonts are embedded in the file. (The font at issue in the case is DecaSerif). As an editor who relies on this in my layouts, it's fairly maddening. I realize this isn't a bug, and that PDF pass-through and access to embedded fonts will come, but for me it can't come soon enough.

So for now I'll take @music2bmade's work-around, and convert my PDFs to TIFF, but I will miss the scalability of the original PDF file.

[MikeW]

1 hour ago, thomaso said:

It did not work for me with any of those from this list. – Also in a code editor app, where I can switch the encoding, I could not make the colons from that pdf appear. – Can you explain a little more about that, @Mike?

I wonder whether it is possible to save such an affected pdf with a different + fitting encoding, to make its glyphs appear correctly in Affinity.
Also, is there a way to get info about the saved and/or the used/shown encoding of a pdf?

p.s.: what do the  2x 3x 4x  in your screenshot stand for?

It's not about the file's encoding, but rather individual glyph's Unicode encoding.

The 2x, etc, has to do with the number of times a glyph appears in a pdf. Because of the way a given font may itself be subsetted in a pdf, this report page will be repeated for each instance, even though technically a single font may be used in the pdf.

[MikeW]

17 minutes ago, stutes said:

I appreciate all the effort that everyone is doing in analyzing my original PDF from BusyCal.

So, again, I appreciate the statement, but if my underlying PDF has a combination of problems, I can't figure out why Publisher is the only app that appears to have a problem with it. All other apps read the PDF just fine.

With all respect, @MikeW , I'd have to agree with @music2bmade. Publisher's inability to read embedded fonts may not be a bug, but Publisher almost seems to be working hard to defeat the purpose of the file format, just for the sake of being able to edit the file type.

Just now this issue reared it's ugly head in a different way: I downloaded a news bulletin to insert into my handout, and since I did not have the fonts embedded in the PDF, Publisher is not able to interpret the PDF correctly, even though the fonts are embedded in the file. (The font at issue in the case is DecaSerif). As an editor who relies on this in my layouts, it's fairly maddening. I realize this isn't a bug, and that PDF pass-through and access to embedded fonts will come, but for me it can't come soon enough.

So for now I'll take @music2bmade's work-around, and convert my PDFs to TIFF, but I will miss the scalability of the original PDF file.

Personally, if I had to use an application incapable of pdf passthrough, I would use an application capable of converting fonts to curves, but I have ripped certain pdfs to images as well. Even for use in layout applications capable of pdf passthrough, like ID & QXP. Ain't nothing wrong with it at all.

And when pdf passthrough is available in APub, and if I used it for a particular job, I would do the same.

[kenmcd]

@stutes

Would you like to test a font which appears to be working?

APub seems to do better with glyphs from OpenType features when the full font is embedded.
So I converted SF to OpenType with TrueType outlines and tested.
I put some of your dates text into LibreOffice and created two PDFs.
- Export to PDF - which embeds a font subset
- Printed to a PDF printer which allows me to force embed the entire font(s).

I expected the subset to not work, and the full embed to work.
Surprisingly it all worked in APub.
Both Placed PDFs appeared correctly (meaning the right glyph appeared).
Opening both PDFs in APub also worked correctly (meaning the right glyph appeared).

I would be interested in seeing your results with the same font.

[stutes]

@LibreTraining, this issue, I'm certain, will rear it's head when I put together my next newsletter in a couple of weeks. I'll try some other fonts then, and will report back. Thanks.

[kenmcd]

35 minutes ago, stutes said:

@LibreTraining, this issue, I'm certain, will rear it's head when I put together my next newsletter in a couple of weeks. I'll try some other fonts then, and will report back. Thanks.

I sent you the test fonts via PM.
The Display-Bold and the Text-Regular like in your original PDF.
The fonts have been converted to OpenType-TrueType which can be embedded fully.

Your original PDF only includes a subset of the fonts.
Embedding the full font may work.
As I mentioned above I have a PDF printer for Windows which allows me to force embed the entire font. So if you could find something similar for Mac that may help.

Be interested to see what happens.

[stutes]

On 9/7/2019 at 5:21 PM, LibreTraining said:

@stutes

Would you like to test a font which appears to be working?

APub seems to do better with glyphs from OpenType features when the full font is embedded.
So I converted SF to OpenType with TrueType outlines and tested.
I put some of your dates text into LibreOffice and created two PDFs.
- Export to PDF - which embeds a font subset
- Printed to a PDF printer which allows me to force embed the entire font(s).

I expected the subset to not work, and the full embed to work.
Surprisingly it all worked in APub.
Both Placed PDFs appeared correctly (meaning the right glyph appeared).
Opening both PDFs in APub also worked correctly (meaning the right glyph appeared).

I would be interested in seeing your results with the same font.

@LibreTraining So today is the day in which I insert the office calendar into our monthly newsletter, and I downloaded your fonts and installed them. My font manager app (Rightfont) didn't seem to like the FriscoProDisplay-Bold, as it never appeared in the font manager, but it did install the FriscoProText-Regular. So I switched the font in my calendar app (BusyCal) to use the FriscoProText-Regular, saved is as a PDF as I normally would through the printer dialogue, and IT WORKED. So thanks for that.

So, to understand correctly, Publisher does better with OpenType fonts?

[kenmcd]

On 9/26/2019 at 8:08 AM, stutes said:

So, to understand correctly, Publisher does better with OpenType fonts?

OpenType fonts with TrueType outlines rather than OpenType fonts with PostScript outlines.

If you do need the Display font I would be happy to help get it working.
Apple sabotaged these fonts in a couple ways.
- the name fields are configured to prevent them working properly on Windows apps
- there is a deliberately incorrect checksum in the fonts which may cause problems

I figured the way I fixed and converted these fonts would circumvent those traps.
It does bother me that the Display font has an issue (a challenge!).
Again, if you do need the Display font I would be happy to help get it working.

Glad to hear it solved part of your problems.
 

[A_B_C]

(I removed my previous post concerning font license issues as I understand that the intention of LibreTraining was to help the original poster.)

[stutes]

@A_B_C, in LibreTraining's defense, he didn't post the reversed engineered font widely on this forum, just in a direct message to me. I was able to read your original post, and I think you bring up some good points. I had originally installed the font to emulate the UI of the phone in a mockup. Clearly my use in calendar is outside the license; I will look into other font options next month. Thanks again for pointing out the license.

[A_B_C]

Thank you for your kind response.

I guess the problem with parsing the PDF exported from your calendar app is precisely the one that @LibreTraining had diagnosed in his or her first response. The San Francisco fonts have a contextual alternates feature that replaces the default colon (Uni+003A) by an alternative version that doesn’t have a Unicode code point assigned. This feature is invoked when the colon is used between tabular or proportional figures.

It would be interesting to see what happens when you switch your calendar app to a font like Source Sans Pro or another open source typeface that doesn’t have the respective OpenType feature. I would venture to guess that your import should work well from the outset. Or at least, it would surely be possible to tweak the typeface such that you don’t have any more problems in the future. 

Thank you again,

Alex

[kenmcd]

Dear Font Police,
The San Francisco fonts are freely available for download from the Apple developer site.
Anyone with the proper tools can modify the fonts, and they do everyday.
A popular fonts sharing social media group has had a full family version of the San Francisco fonts available for quite some time which has been fixed to work properly on Windows by fixing the Apple names sabotage.
I am sure they have only been downloaded a few hundreds, or thousands, of times.
For development purposes only, for sure.
Perhaps some font policing skills are needed there.

Good thing Apple has self-appointed holier-than-tho font police to protect their meager fortunes from attack by unscrupulous operators seeking to understand and deal with development and editing issues related to PDF import of an OpenType raised colon.
I am sure Apple will be sending their thanks soon for protecting them from this evil affront.
<roll-eyes>

Why is it every development forum has at least one font police protector-of-the-license?
No font creator with more than half a brain is going to give a rats ass about people using their fonts to try and figure-out issues in an up-and-coming graphic design application.
They would have to be a complete moron.