Why is a simple Data Merge so painfully slow in Publisher?

[Dero]

I'm now sitting here for 10 minutes waiting for ~7,500 purely text entries to be merged into a simple grid.

This is in Publisher 2.5.5 (Windows 10).

What is going on?

My source document is a simple 2 column, ~7,500 row CSV document. The template I'm merging into is a 47×3 data merge grid, each cell containing two text fields and a line. How does this operation take over 10 minutes on a high end desktop PC?

It now finished, 15 minutes later. It generated 54 pages and the result is correct, so I don't think this is a skill issue on my part.

Attaching source files in case anyone wanted to look into this with the exact assets I'm using.

Thanks!

library-template.zip

[lacerto]

Yes, I can reproduce this. I suspected that this could somehow be related to your having unnecessary field separator at the end of each record, or that the Unicode encoding could affect this, or having artistic text fields instead of frame text fields, but I changed the names to "John Doe" and saved the file as a regular Excel file, made text field changes, etc., but the same slowness could be experienced also with a "fixed" configuration. I am not sure if this is something "new": it is likely that I have not earlier tried this feature with thousands of records. I have earlier noticed some initial slowness in processing ("analyzing" stage), but typically merging is finished pretty fast after that. 

[thomaso]

I can't open V2 so I rebuild from its preview in V1. The generating took ~90 seconds.
(I run it twice because first I was confused by the rows per column, until I noticed the missing numbers in the CSV, e.g. 31, 41, 42, 43)

datamerge 1 layout.afpub   datamerge 2 result.afpub

[lacerto]

I already doubted that Apple has sponsored Serif with this, but on a new M3 Mac this took about 10 minutes (on my new Windows laptop, the same price, about 15 minutes) -- oops, now I promoted the Mac sales myself... So something in data merge got broken with version 2.

[Old Bruce]

You have three lines with over a hundred characters in the second field. I wonder if that might be part of the problem.

[Dero]

21 minutes ago, Old Bruce said:

You have three lines with over a hundred characters in the second field. I wonder if that might be part of the problem.

I can't image why that would affect it in any way. Longer text could require a little bit more layouting and rendering time, but since it's constrained to a single cell, that extra time should be virtually imperceptible.

1 hour ago, lacerto said:

Yes, I can reproduce this. I suspected that this could somehow be related to your having unnecessary field separator at the end of each record, or that the Unicode encoding could affect this, or having artistic text fields instead of frame text fields, but [...]

Thank you for testing this on both platforms and for testing with a more "bare-bones" example! It would surprise me if any of those aspects would affect rendering time so severely. There is something very broken in v2 of Publisher. Interpolating ~7500 (times two) text strings into a basic layout should take a few seconds at most.

[wonderings]

I did a comparison doing the merge in Pub 2 and Indesign. Both latest updates. The test was not very scientific as I did not use a stop watch or anything, but Publisher did it in around 7 or 8 minutes on my M1 Max with 32 gigs of RAM. Indesign did it in just under 30 seconds (I did use a stopwatch this time). Quite the performance difference. I have never really played around with the data merge, but this difference in performance is startling and would certainly be an issue when doing 10,000+ page merges with full graphics. 

[walt.farrell]

There seems to be something nonlinear in the time it takes.

For that data and layout:

• 1000 records: 0:43 (m:s)
• 2000 records: 2:00
• 3000 records: 3:54

I will note that each element of the Data Merge Layout ends up generating a Group, which contains a rectangle, a curve, and two Artistic Text objects. So that's 1 group with 4 objects per record, or roughly 29,600 objects for the entire run.

Also, I will note that these timings were done with Preflight Checking disabled and the Spelling/Hyphenation languages for the fields set to None, in case that has some influence.

[lacerto]

I think that the test made by @thomaso is enough to show that something got changed between the versions. I, too, noticed the non-linearity in processing, but the number of generated objects in the result is the same as in v1 run, so while the increase in the amount of objects to process does have an effect in v2 performance, it cannot directly explain [but e.g. a memory leak in processing could] the difference in run times between the versions. The timings shown in context of InDesign are more in line with what might generally be expected when running such a simple task. 

[thomaso]

12 hours ago, lacerto said:

I, too, noticed the non-linearity in processing, but the number of generated objects in the result is the same as in v1 run

FWIW, I added in my V1 layout the horizontal underlines (as curve) which increased the generation time from 90 to 120 seconds, which means + 33% just for the lines!

I also noticed, other than expected, that APub's Performance settings don't influence the generation time, I get the same duration for these settings…

… where RAM and CPU usage in ActivityMonitor.app did not rise above 101.6% for either setting (a relative value, not an absolute one), which seems to mean that data merge generation is not particularly demanding on RAM or CPU.

[lacerto]

Ok, addition of graphic resources like lines might result in memory leaks (e.g. errors in creating and failure to release device contexts), which would typically manifest as something like this. 

UPDATE: This was not confirmed in tests that I ran with version 1:

3:21 with a line

2:43 without a line

2:52 line applied as a single Curves object on master page

...so basically the line is just one object more when it is part of a data merge layout composition. No indication of memory leak or gradual slowing down, or anything like that. Processing on a PC is slower than on a Mac, and about 4 times slower in version 2, compared to version 1.

UPDATE2:

7:19 Optimized with lines on a master page and text fields in a single frame text object; version 2, Windows

1:25 The same job done with version 1, Windows

[Lee D]

@Dero I've passed over your sample files to our developers to investigate further.

[Dero]

Thanks, @Lee D, appreciated!