Data Merge unusable on large quantities, insanely long export time and enormous amount of disk space is consumed

[Bruno Belo]

21 minutes ago, Old Bruce said:

So you have two fields; a "number" and a "white page". I understand what the number is but I am struggling to figure out what the white page is. What is in the Excel file for that cell? In the screen show (of Apple's Numbers) what would be in the cells with the "???".

Nothing on column B, only column A. 

The white page is a blank page only with the number. Example:

Page 1 generated after data merge have only number 5001 but twice ( two fields = same field repeateadely)

The job is a ticket, and i need to generate 13500 ticket numbers 

 

[Bruno Belo]

21 minutes ago, Old Bruce said:

So you have two fields; a "number" and a "white page". I understand what the number is but I am struggling to figure out what the white page is. What is in the Excel file for that cell? In the screen show (of Apple's Numbers) what would be in the cells with the "???".

Nothing on column B, only column A. 

The white page is a blank page only with the number. Example:

Page 1 generated after data merge have only number 5001 but twice ( two fields = same field repeateadely)

The job is a ticket, and i need to generate 13500 ticket numbers 

 

[Bruno Belo]

8 minutes ago, Bruno Belo said:

Nothing on column B, only column A. 

The white page is a blank page only with the number. Example:

Page 1 generated after data merge have only number 5001 but twice ( two fields = same field repeateadely)

The job is a ticket, and i need to generate 13500 ticket numbers 

 

the question is that I manage to generate the 13500 "tickets" but the afp generates a file with 13,500 pages (there is no problem with that, it is correct), the problem is consuming 25gb of ram and when going to the export option the program closes.

I can't export because the menu doesn't even appear, the program closes. I have 32gb of ram and plenty of space on an nvme ssd, integrated graphics card amd 5600g and dedicated geforce gtx660, windows 10. a good machine.

[Old Bruce]

For Tickets I would use Publisher's Flow options in a Paragraph Style.

Make a text frame on a Master Page for the Ticket Number. Copy your numbers (5,001 to 13,000 or whatever) and paste it into the Ticket number Text Frame (currently empty in the For Ticket Numbers file. Use the Select all to select all the text, and apply the Ticket Number Style . Now Shift + Click on the little overflow indicator and you'll get new pages each with only one number. Paste the same numbers into Page 1's other Ticket number frame and shift click on that overflow indicator and you'll get the second copy made.

for ticket numbers.afpub

for ticket numbers finished.afpub

[Bruno Belo]

20 minutes ago, Old Bruce said:

For Tickets I would use Publisher's Flow options in a Paragraph Style.

Make a text frame on a Master Page for the Ticket Number. Copy your numbers (5,001 to 13,000 or whatever) and paste it into the Ticket number Text Frame (currently empty in the For Ticket Numbers file. Use the Select all to select all the text, and apply the Ticket Number Style . Now Shift + Click on the little overflow indicator and you'll get new pages each with only one number. Paste the same numbers into Page 1's other Ticket number frame and shift click on that overflow indicator and you'll get the second copy made.

for ticket numbers.afpub

for ticket numbers finished.afpub

That's cool, thanks, nice to know there's this option. But I really wish we had a solution for data merging, because what I try to do is very simple, I could do it in MS Word, WPS Office, or Indesign too.

If this is something that AFP has trouble doing it would be nice for the team to know about it, as generating a certain amount of simple data with nothing applied as a background seems too light to me, maybe just needing refinement in the programming.

I believe in the potential of affinity and I use data merge where I work to make badges, coupons, numbered blocks, certificates and everything possible using variable data, but I use indesign most of the time. Take my attached file to test it out. Thanks

base.afpub Pasta de Trabalho1.xlsx

[Bruno Belo]

13 minutes ago, Bruno Belo said:

That's cool, thanks, nice to know there's this option. But I really wish we had a solution for data merging, because what I try to do is very simple, I could do it in MS Word, WPS Office, or Indesign too.

If this is something that AFP has trouble doing it would be nice for the team to know about it, as generating a certain amount of simple data with nothing applied as a background seems too light to me, maybe just needing refinement in the programming.

I believe in the potential of affinity and I use data merge where I work to make badges, coupons, numbered blocks, certificates and everything possible using variable data, but I use indesign most of the time. Take my attached file to test it out. Thanks

base.afpub 21.01 kB · 0 downloads Pasta de Trabalho1.xlsx 110.59 kB · 0 downloads

I did another test now, I lowered the amount to 5,000 numbers. By the way, I did two tests, one with 5,000 numbers and nothing in the background of the page. And another with 5,000 numbers and a pdf imported to the background of the page ( linked ), a scanned pdf with 5mb.

The 5,000 without the background applied generated normally and I was able to export. The 5,000 with the pdf in the background generated but the program closed when trying to export. Both had a consumption of 11 to 12gb of RAM in the generated file with 5,000 pages.

[Blake_S]

Something goes horribly wrong when AFP creates a merged file.

Just now a merged file with 412 pages, all having the same linked PDF from 2 master pages and a single text field on each page, has a 1420 MB (1.4 GB) size after being saved. I'm not able to export it, errors out every time.
This is stupid. It should be under 10 MB.
The linked PDF size is 1 MB and it contains 2 pages.
Besides the links to that PDF and text fields with personalised data, there is nothing else.

This suggests that AFP is doing something stupid, like creating a separate copy of a linked file for each page would be my guess.

Looks like its back to InDesign CS5 for good for me until this gets resolved.

___________________

Update - re-created exact same job in InDesign CS5.
Export took about 5 seconds, vs. AFP erroring out on export after like 3 or 4 minutes.

[Bruno Belo]

10 hours ago, Blake_S said:

Something goes horribly wrong when AFP creates a merged file.

Just now a merged file with 412 pages, all having the same linked PDF from 2 master pages and a single text field on each page, has a 1420 MB (1.4 GB) size after being saved. I'm not able to export it, errors out every time.
This is stupid. It should be under 10 MB.
The linked PDF size is 1 MB and it contains 2 pages.
Besides the links to that PDF and text fields with personalised data, there is nothing else.

This suggests that AFP is doing something stupid, like creating a separate copy of a linked file for each page would be my guess.

Looks like its back to InDesign CS5 for good for me until this gets resolved.

___________________

Update - re-created exact same job in InDesign CS5.
Export took about 5 seconds, vs. AFP erroring out on export after like 3 or 4 minutes.

Yes, i use data merge in long years, ms word, corel draw, indesign, planepress, ok some of programs are specific for this, but afp is in begin of this function.

Afp is a incredible software and i love it but data merge need more attention.

Out of data merge, i import 456 pdf pages linked from a pdf with less then 5mb. The use of ram reaches more than 20gb, i sucessfull export when i set the performance limit use of ram to afp to the max possible ( 65gb ) using 20gb ram of system and rest of virtual memory, only method to work.

 

[Blake_S]

So, will this be addressed? Or do we just give up on using the Data Merge function in Affinity?

Bug is still present in the latest version. Just now tried to data merge a simple ticket with only 600 numbers, which are placed into only one text field.
A 1.8 MB PDF is used as the background.
Realistically, this should take less than 5 seconds to merge.

After one minute and the drive running out of space because Affinity managed to write 40GB of data, I killed it via task manager and went back to InDesign CS5 to do the job instead...

As a comparison, InDesign CS5 took 4 seconds to export 600 records, 14 seconds for 3000 records, and didn't write any noticeable amount of data to the disk.

[werfox]

Maybe the developers will start to work on this issue:

in this post, Dan C (staff member) said that the behaviour might be "as expected", but it might be considered a bug AF-3294:

 

On 6/3/2024 at 10:27 AM, Dan C said:

I can confirm I've been able to replicate this with your files provided and therefore have logged this with our development team for you now.

However I'm not 100% certain if this is classified by our team as a bug or an improvement request, as I believe the current behaviour of Publisher may be expected, but as the behaviour from InDesign is much more sensible where applicable, therefore it is now with our team for further considerations.

I hope this helps

I had a glance at the documentation of the used PDFlib, there should even be a possibility to incorporate a real PDF-VT output (where the described problems with huge file sizes should not occur), but this is up to the developers.

 

Best regards

[Dan C]

25 minutes ago, werfox said:

Maybe the developers will start to work on this issue:

in this post, Dan C (staff member) said that the behaviour might be "as expected", but it might be considered a bug AF-3294:

To confirm, the specific bug I have raised with our development team (AF-3294) relates to the duplication of placed files being exported for each instance within a PDF file, increasing the size of the PDF file unexpectedly.

For example if you were to create a document with 1 page and place 4 files onto this page, it would export to PDF at a hypothetical 10Mb.
If you then duplicate this page in Affinity 10 times and export to PDF once again, the size of the exported PDF increases almost 1-for-1 with the page number increase, for a hypothetical 100Mb exported file.

However other applications, such as InDesign, are able to intrinsically 'link' duplicated elements within an exported PDF, drastically reducing the exported file size - and this is what the development log is requesting of Publisher in a future update.

30 minutes ago, werfox said:

there should even be a possibility to incorporate a real PDF-VT output

I'm not personally au fait with this section of the PDF standard, but from a brief overview I believe PDF/VT support would be slightly different to the above report (but perhaps one possible solution to it) and therefore I'd recommend posting a new thread in the Feedback section requesting support for this standard, as I understand we would not class it's current non-inclusion as a bug.

I hope this clears things up

[werfox]

Hi @Dan C,

thanks for the explanation. I’ll post a request for PDF/VT as suggested.
And here it is:

Quote

 

 

Best regards.

Edited July 4, 2024 by werfox
link to feature request added

[Dazmondo77]

I set tickets up on an SRA3 sheet and auto number them via Bullets and numbering in the paragraph panel which works really well see vid:

 

 

[Affinity Info Bot]

The issue "PDF export individually includes duplicated placed files unexpectedly, causing larger than necessary files" (REF: AF-3294) has been fixed by the developers in the latest beta build (2.6.2.3213). The fix is planned for inclusion in the next customer release.
Customer beta builds are announced here and you can participate by following these instructions.
If you still experience this problem once you are using that build version (or later) please reply to this thread including @Affinity Info Bot to notify us.