Enormous RAM consumprion/disk write during data merge

[Blake_S]

On a data merge operation Publisher consumes all available memory, then proceeds to write 30-40GB to disk in a PersonaBackstore.dat just to make 500 record merge that when exported has a size of 3.5 MB

It also takes around 2 minutes to process.

Test was done with a 1-page document with a 2MB PDF placed on the master page and applied to that one page,
all data merge fields were text only and did not influence the results.

Using different PDF files also did not influence the results in any significant way.
Checking the PDF files used with Preflight of Acrobat Pro shows no errors.
PDF objects use CMYK color space.

Placing PDF directly on the page or placing it on the master page and applying that master page to the regular page did not influence the results.

Amount of data written to disk is dependent on the number of records processed:

250 records - 14,6 GB written
515 records - 33 GB written
750 records - 59.5 GB written

 

[walt.farrell]

For previous discussion:

 

[Blake_S]

Using "Update" button in the Data Merge Manager seems to fix this issue,

after that the merge takes <5 seconds, nothing is written to disk.

If I don't save this afpub file, re-open it, and attempt to do the merge, then 2 minute processing time + 30GB of data written returns, unless I click "Update" again.

Only manually clicking the button works.

If I first click Generate, then Publisher presents an option to update data sources before merging, and I confirm the update, this does not work - then it does the slow 2 minute merge.

 

[Old Bruce]

6 hours ago, Blake_S said:

If I first click Generate, then Publisher presents an option to update data sources before merging, and I confirm the update, this does not work - then it does the slow 2 minute merge.

Sounds like a bug.

I would delete the data source and load it back in, save the file, then try again.

[Lee_T]

Just an update, we're still investigating this, there's obviously something not quite right.

Lee

[Leigh]

On 9/16/2022 at 1:31 PM, Blake_S said:

Using "Update" button in the Data Merge Manager seems to fix this issue,

after that the merge takes <5 seconds, nothing is written to disk.

If I don't save this afpub file, re-open it, and attempt to do the merge, then 2 minute processing time + 30GB of data written returns, unless I click "Update" again.

Only manually clicking the button works.

If I first click Generate, then Publisher presents an option to update data sources before merging, and I confirm the update, this does not work - then it does the slow 2 minute merge.

 

Thanks for the update on this.

We're still investigating but I wanted to give you an update - Lee is currently out of the office.

I've been able to replicate the issue with the files you sent to Lee via Dropbox. I'm pretty sure it's related to the PDF on the Master Page - when I delete the first group containing the clipped curve layer and the cut marks it appears to merge without using excessive memory and doesn't page to disk.

I've tried the steps you mention above but that didn't resolve the issue for me. Would it be possible to send a screen recording of you doing the steps? Also, were you using the same linked PDF which was sent to Lee or a different PDF?

[Blake_S]

I can confirm that I can no longer fix the issue with the test project that I sent by just clicking Update button.

However, I've run into this bug again on a new data merge job, with different files.

First data merge went nearly instantly for 500 records.

However, replacing a PDF on a master page with a new one via Resource Manager triggered the bug, resulting is slow merge and huge memory consumption.

When I deleted said PDF from the master page, and just placed the new PDF file there manually, not involving Resource Manager, this fixed the issue.

[Leigh]

Thanks for this additional information - will give this a try. Did any of the new PDFs you used in this new job have crop marks at all?

[Blake_S]

2 hours ago, Leigh said:

Did any of the new PDFs you used in this new job have crop marks at all?

no, they don't have any crop marks or any other kinds of marks

[Leigh]

16 hours ago, Blake_S said:

no, they don't have any crop marks or any other kinds of marks

Ok, thanks! We'll try the steps in your previous post and let you know the outcome.

[Leigh]

Morning @Blake_S Just wanted to let you know that our QA team could also replicate the issue but have been unable to figure out the cause of the issue. For now, continue to use your workaround and I will let you know if we manage to find a better answer/solution to the issue.

Thanks again for letting us know 👍