Add Markdown file support

[Frank Jonen]

This should've been in the previous release but here it is again. Markdown import so we don't have to convert to legacy formats or use crude copy & paste for each article.

Solution: Link Markdown files to text frames like the data merge preview.

That way the documents can be worked on by writers/editors and kept up-to-date via the Resource Manager.

[PaoloT]

While doing some work created as a markdown file, and then opened with a page layout program as an IDML file, I want to add my vote to this feature again.

There are two different mindsets, when preparing a book:

a) that of those who see the book as a graphic work right from the beginning; and

b) that of those who see it as a flow of information, that will only in the very end get its final shape as a printed page, or anything else.

My work always requires the latter mindset, and it's a joy when you see all the preliminary work flourishing as a page in Publisher. Ah, if only it was easier to go from markdown to page…

Paolo

 

[ben_zen]

I'm writing a novel currently, using Markdown as my writing format; I want to produce text right now, I don't want to deal with formatting beyond declaring that something's italicized, bolded, whatever. When it comes to page layout, however, while I'm familiar with LaTeX and wouldn't be opposed to exporting to that and using it to generate PDFs, I was hoping to use Publisher, since I've enjoyed Designer so much. Lacking direct Markdown import means I'd need to export first to Word, then import; this also would break my flow for edits, since my root text is always going to be the Markdown.

Part of the goal here is to retain the structural clarity that markdown provides, and the simplicity of working with it, while getting the control of a proper publishing suite.

[Frank Jonen]

4 hours ago, ben_zen said:

while I'm familiar with LaTeX

You might like Typst and since Pandoc 3.1.2 there's a path from Markdown to Typst. It's a bit easier for hired editors to work in because of the browser based UI. The template language has CSS ideas as its base.

Affinity probably plans to do what Amazon does "…we're seeing an increase in requests for Markdown support. … In the meantime you can use the export function of your Markdown editor to create Microsoft Word compatible files…" Basically try to wait it out for several years like Microsoft did with the Internet.

 

[Old Bruce]

Note: I am not a Markdown user.

Can we not use the (plain text) text file from Markdown (that is if there is one) and then run a regular expression find and replace in Publisher to assign Paragraph Styles?

It would help if I could see a Markdown (plain text) text file.

[ben_zen]

17 hours ago, Frank Jonen said:

You might like Typst and since Pandoc 3.1.2 there's a path from Markdown to Typst. It's a bit easier for hired editors to work in because of the browser based UI. The template language has CSS ideas as its base.

I'll definitely give that a look when it comes to editing; I'm happy with my current setup (I'm using Typora for drafting, since all I want is something that puts text on the screen and has some word processing features, and it has a relatively robust pandoc integration.) As for Affinity, I haven't used the suite enough yet (I'm a neophyte Designer user, now looking to extend into Publisher) to really see the limitations on what it handles. I'll probably fall back to the Word export, since that retains the style information I want.

1 hour ago, Old Bruce said:

Note: I am not a Markdown user.

Can we not use the (plain text) text file from Markdown (that is if there is one) and then run a regular expression find and replace in Publisher to assign Paragraph Styles?

It would help if I could see a Markdown (plain text) text file.

I wasn't aware that find & replace could do paragraph styles! That would almost be the way I'd want to do this, although I'd rather it were, well, automated. Here's the CommonMark spec, but a brief example would be something like this:

# Demo
This is just a quick document to show off how **Markdown** is written. It's a simple syntax, really.

## Headings
Headings are defined by lines starting with one or more `#` characters, followed by a space; the _level_ of the heading is determined by how many
hashes there are.

## Lists
Lists are just:
* one thing
* followed by another
  - and can be nested
  - when it makes sense

Gruber originally developed it to be something that could readily convert into HTML, and it's fairly easy to parse. The downside of that is that everyone has their own way of doing it, and not all of them are compatible.

[v_kyr]

1 hour ago, Old Bruce said:

Can we not use the (plain text) text file from Markdown (that is if there is one) and then run a regular expression find and replace in Publisher to assign Paragraph Styles?

That's basically what a bunch of Markdown editors (their parser parts) for a WYSIWYG preview do. The only problem with Markdown is, that there are a bunch of different markdown definition flavors, so also some enhanced Markdown syntax versions/variations.

I've also once (years, aka long time ago) wrote a Markdown editor, whose parser part was heavily based on reg expr usage ...

... which used pretty much the same techniques (even GUI wise) as an even older BB editor (Bulletin board/forum editor) I once created before ...

[PaoloT]

20 hours ago, Frank Jonen said:

You might like Typst and since Pandoc 3.1.2 there's a path from Markdown to Typst.

It's what I’m working on now. It's a fascinating and powerful system, but lacking a lot from what a modern page layout program can do. And, obviously, not exactly user-friendly.

For example, unless I’m happy with a few templates for scientific articles, I have to build my own template by coding it. Typist is a lot easier than LaTeX, and a lot more modern in allowing system fonts and diacritic marks without heavy patching. Still, it's an arcane way of working.

There is no easy way that I’ve found to create object styles. Paragraph and character styles, unless one uses a couple of them, is a hell to type. And multiline cells in tables are still considered something odd. (Thank you to Scrivener for doing mosto of the heavy-lift, here…).

Being able to author in Markdown and publishing with Affinity Publisher would be a dream. But there is no easy path. The Word format used for interchange doesn't seem to allow linked images. IDCL is very basic, and then Publisher wouldn't understand it, unless you manually convert it to IDML. And Markdown is totally ignored. What a shame!

Paolo

 

[Frank Jonen]

7 hours ago, Old Bruce said:

Can we not use the (plain text) text file from Markdown (that is if there is one) and then run a regular expression find and replace in Publisher to assign Paragraph Styles?

I've done that in the past a few times but it's really bad in Publisher. The end result was so bad that I ended up skipping Publisher altogether for that project. RegEx just isn't well supported for automated formatting. Removing the markdown syntax is another thing where Publisher caused errors. 

It's a lot easier to write a basic Markdown parser in Rust than to get Publisher to behave properly with RegEx. The time spent isn't worth the result.

[ben_zen]

8 hours ago, PaoloT said:

It's what I’m working on now. It's a fascinating and powerful system, but lacking a lot from what a modern page layout program can do. And, obviously, not exactly user-friendly.

For example, unless I’m happy with a few templates for scientific articles, I have to build my own template by coding it. Typist is a lot easier than LaTeX, and a lot more modern in allowing system fonts and diacritic marks without heavy patching. Still, it's an arcane way of working.

This is a lot of what has me iffy about using something like Typst. I'm expecting (I'll test this workflow first, of course) that I can find a browser-based solution for whoever I work with on editing, and they'll still be able to hand me whatever edits.

As for how I'd want to use Markdown, it strikes me that, were there direct support, it'd be cool to be able to use the YAML front matter extension to essentially do form-fill. Chapter titles, other elements (time, location) could be stored there, instead of as headings.

[Frank Jonen]

18 hours ago, ben_zen said:

As for how I'd want to use Markdown, it strikes me that, were there direct support, it'd be cool to be able to use the YAML front matter extension to essentially do form-fill. Chapter titles, other elements (time, location) could be stored there, instead of as headings.

The thing is, this is really easy (I wrote a basic parser that uses frontmatter). Publisher has all the building blocks for it. Yet for some reason they keep deciding against it.

My experiment with MD in in Publisher was leveraging that. I used the data import for JSON files with veeeery long strings. Each section was stored in its own string value and each chapter as a separate file. The Affinity RegEx limitations ruined it though. You can't automate anything, you have to launch each element replacement and each markup deletion manually for each element group. Takes forever and is error prone. Hence "not worth the effort".

[ben_zen]

Yeah, that makes sense. I'd really rather they had the native support, than have to deal with regexes (my current bane elsewhere, too.)

Well, hopefully they'll consider it, because I'd really love a direct workflow like that.

[PaoloT]

I was thinking that an Markdown importer/exporter would not only be useful to move data (from blogs, Scrivener, or any text editor) into Publisher, but also the opposite. Create a traditional page layout in Publisher, and then export to Markdown, to reuse it for web or other formats. A use case might be this one:

1. Write an article, report or book with any text editor (maybe, starting from a series of separate blog posts).

2. Import the Markdown file into Publisher, and do all the necessary editing for making it look great.

3. Ask your reviewers to check the PDF, and finish your publication.

4. Export your project from Publisher to Markdown.

5. Use Pandoc or Quarto to make a Web page, a Website, or an HTML Book or it.

Converting text styles from/to Markdown tags would be easy.

Images will have to be linked, and Publisher would just have to replace them with the real images when importing, or exporting the linked images to Markdown links when exporting. Width and height will also be easy to convert. Object/image styles – when Publisher will support them – will become Markdown classes, and vice-versa.

While it is clear that IDML is a very important compatibility for having Publisher accepted in a traditional publishing workflow, Markdown would immediately open it to the future, that is already the present: multi-channel publishing, not mainly targeted to printed publications, but to the web.

Incidentally: unless I'm missing something, this would be an advantage of Publisher over InDesign.

Paolo

 

[MikeW]

There would need to have a means for style mapping even if one was really strict about style naming when authoring in APub first. Just because the ideal workflow is MD first doesn't mean it is going to happen. 

We need style mapping anyway for importing from Word, too.

Tables would be a challenge to represent all one can do in a layout application going into MD.

[PaoloT]

54 minutes ago, MikeW said:

There would need to have a means for style mapping

In modern markdown (I'm mostly considering Pandoc and Quarto's dialects), you can create divs and spans, and add classes to them. This is like adding styles to paragraphs and portions of text. For example, you can have divs like this:

::: {.Warning}
Text.
:::

where the class "Warning" is equivalent to a paragraph style named "Warning".

And you can have spans like this:

This is some [special text]{.Emphasis}

where the class "Emphasis" is equivalent to a character style named "Emphasis".

How these are visually interpreted depends on the layout program. Publisher or Word will have to apply their style sheet from a template.

54 minutes ago, MikeW said:

Tables would be a challenge to represent all one can do in a layout application going into MD.

Tables have limits in markdown, but complex tables can be represented. Column alignment can be set even in the basic markdown. Column width can be set by percentage in Pandoc and Quarto (and this is the most significant measurement, for a language mainly conceived for the web). I sincerely ignore if tables can be set by exact measurement. I don't even know if this can be indicated via CSS (that could be mixed with markdown).

Paolo

 

[MikeW]

Ok. Thanks for that. 

So is there a universal WYSIWYG authoring application, at least in the sense of Mac/PC, that supports classes et al that you are demonstrating?

Or are you proposing Serif handle the half million flavors of MD and or their extensions for both import and export?

[MikeW]

btw, I started to receive very simplistic MD from one publisher a few years ago. There are, at this point, only 16 formatting-related attributes. I just keep adding conversion strings to a JS in my text editor to convert the text to QXP (well, Em Software's flavor) tagged text. That in turn is imported to a template and is formatted, with the exception of tables. Those are stripped of the MD and tabs added. Presentational attributes are done directly in QXP via converting the tabbed text to a table, then a table style applied.

So this topic does have relevance, despite my ignorance of the various MD "languages" and/or their extended varieties.

[PaoloT]

2 hours ago, MikeW said:

So is there a universal WYSIWYG authoring application, at least in the sense of Mac/PC, that supports classes et al that you are demonstrating?

Or are you proposing Serif handle the half million flavors of MD and or their extensions for both import and export?

The MD editors I know (VS Code, Ulysses, Byword, iA Writer, Editorial) have very limited WYSIWYG support. In any case, I'm not suggesting that Publisher becomes an MD editor, but that its files could be converted to MD, inside the limits of how much can be converted. Conversion from MD shouldn't be all that a problem, since Publisher can do more than MD.

Markdown flavors are not so many, now. There is convergence toward CommonMark, already accepted by Microsoft, and jointly developed by the creators of Pandoc, GitHub, Reddit, Discourse, and others. Admittedly, there is Quarto, trying to fill some voids, but also influencing the CommonMark board and likely soon part of it. In any case, anything Pandoc can understand, Quarto can as well.

Markdown is often misunderstood, being considered just a simplistic thing. On the contrary, the brilliant intuition of John Gruber was that something as complex as HTML could be represented with human-readable language. It should be relevant to anybody doing some sort of writing and publication in these times.

Paolo

 

[v_kyr]

The initial one from 2004 tells the motivation about it ...

• https://web.archive.org/web/20040402182332/http://daringfireball.net/projects/markdown/

[MikeW]

2 hours ago, PaoloT said:

... There is convergence toward CommonMark, ...

If CommonMark support is added, I think that people will want various extensions supported too. It's pretty bare bones. That is pretty much the format I receive and turn into tagged text. 

For output, would html5 suffice? It has the ability to present what APub can accomplish and more. If html5 can then be converted into other formats, it would be good, yes?

[v_kyr]

40 minutes ago, MikeW said:

For output, would html5 suffice? It has the ability to present what APub can accomplish and more. If html5 can then be converted into other formats, it would be good, yes?

Isn't it obvious that once you can parse in some file format, like MD, and transform it into the Affinity internal format structure, that you then would be able to export to all it supports to generate? - And of course being able to exporting also as HTML5 would make much sense here, as it either way goes hand in hand for representing markdown in a WYSIWYG manner in any webbrowser or app webviews nowadays.

[MikeW]

54 minutes ago, v_kyr said:

Isn't it obvious that once you can parse in some file format, like MD, and transform it into the Affinity internal format structure, that you then would be able to export to all it supports to generate? - And of course being able to exporting also as HTML5 would make much sense here, as it either way goes hand in hand for representing markdown in a WYSIWYG manner in any webbrowser or app webviews nowadays.

Yes, but isn't it equally obvious that no current flavor of MD supports APub's current capabilities, much less future capabilities. Meaning, it's one thing to support the simplistic CommonMark MD format coming in...but once "fleshed out" in APub it seems silly to then export it back out in that format as there could/would be significant loss (at least representational loss). For instance, even if APub went beyond CommonMark and supported some flavor of imported MD tables, once in APub, cells combined, etc., export to MD would be quite a mess.

Which is why I suggested html5 as a viable representational output.

Anyway, I doubt I'll see significant use of MD before I (really) retire as regards needing to do something other than the conversion to tagged text that I currently do.

[v_kyr]

8 hours ago, MikeW said:

Meaning, it's one thing to support the simplistic CommonMark MD format coming in...but once "fleshed out" in APub it seems silly to then export it back out in that format as there could/would be significant loss (at least representational loss).

Who wants this? MD is/was meant first of all to be an easier to write down and being readable textual input format which then can be transformed into XHTM/HTML. It isn't usually meant to be a full blown output replacement format for other more complex file formats. So it's overall usage was more meant to be like troff/nroff/TeX/LaTeX here (plain text with some annotated format tags) and not to be auto-generated out of other more complex file formats.

 

[MikeW]

2 minutes ago, v_kyr said:

Who wants this? MD is/was meant first of all to be an easier to write down and being readable textual input format which then can be transformed into XHTM/HTML. It isn't usually meant to be a full blown output replacement format for other more complex file formats. So it's overall usage was more meant to be like troff/nroff/TeX/LaTeX here (plain text with some annotated format tags) and not to be auto-generated out of other more complex file formats.

 

@PaoloT for one:

https://forum.affinity.serif.com/index.php?/topic/174290-add-markdown-file-support/#comment-1132783

I was, had been, trying to convey that really isn't going to happen no matter the MD format/extension...hence asking Paolo if html5 export would suffice.

[v_kyr]

1 minute ago, MikeW said:

hence asking Paolo if html5 export would suffice

HTML5 export would be fine for that, and either way having HTML5 output support too (independently from MD input support here) would be generally good to have.