Data Merge unusable on large quantities, insanely long export time and enormous amount of disk space is consumed

[Bruno Belo]

21 minutes ago, Old Bruce said:

So you have two fields; a "number" and a "white page". I understand what the number is but I am struggling to figure out what the white page is. What is in the Excel file for that cell? In the screen show (of Apple's Numbers) what would be in the cells with the "???".

Nothing on column B, only column A. 

The white page is a blank page only with the number. Example:

Page 1 generated after data merge have only number 5001 but twice ( two fields = same field repeateadely)

The job is a ticket, and i need to generate 13500 ticket numbers 

 

[Bruno Belo]

21 minutes ago, Old Bruce said:

So you have two fields; a "number" and a "white page". I understand what the number is but I am struggling to figure out what the white page is. What is in the Excel file for that cell? In the screen show (of Apple's Numbers) what would be in the cells with the "???".

Nothing on column B, only column A. 

The white page is a blank page only with the number. Example:

Page 1 generated after data merge have only number 5001 but twice ( two fields = same field repeateadely)

The job is a ticket, and i need to generate 13500 ticket numbers 

 

[Bruno Belo]

8 minutes ago, Bruno Belo said:

Nothing on column B, only column A. 

The white page is a blank page only with the number. Example:

Page 1 generated after data merge have only number 5001 but twice ( two fields = same field repeateadely)

The job is a ticket, and i need to generate 13500 ticket numbers 

 

the question is that I manage to generate the 13500 "tickets" but the afp generates a file with 13,500 pages (there is no problem with that, it is correct), the problem is consuming 25gb of ram and when going to the export option the program closes.

I can't export because the menu doesn't even appear, the program closes. I have 32gb of ram and plenty of space on an nvme ssd, integrated graphics card amd 5600g and dedicated geforce gtx660, windows 10. a good machine.

[Old Bruce]

For Tickets I would use Publisher's Flow options in a Paragraph Style.

Make a text frame on a Master Page for the Ticket Number. Copy your numbers (5,001 to 13,000 or whatever) and paste it into the Ticket number Text Frame (currently empty in the For Ticket Numbers file. Use the Select all to select all the text, and apply the Ticket Number Style . Now Shift + Click on the little overflow indicator and you'll get new pages each with only one number. Paste the same numbers into Page 1's other Ticket number frame and shift click on that overflow indicator and you'll get the second copy made.

for ticket numbers.afpub

for ticket numbers finished.afpub

[Bruno Belo]

20 minutes ago, Old Bruce said:

For Tickets I would use Publisher's Flow options in a Paragraph Style.

Make a text frame on a Master Page for the Ticket Number. Copy your numbers (5,001 to 13,000 or whatever) and paste it into the Ticket number Text Frame (currently empty in the For Ticket Numbers file. Use the Select all to select all the text, and apply the Ticket Number Style . Now Shift + Click on the little overflow indicator and you'll get new pages each with only one number. Paste the same numbers into Page 1's other Ticket number frame and shift click on that overflow indicator and you'll get the second copy made.

for ticket numbers.afpub

for ticket numbers finished.afpub

That's cool, thanks, nice to know there's this option. But I really wish we had a solution for data merging, because what I try to do is very simple, I could do it in MS Word, WPS Office, or Indesign too.

If this is something that AFP has trouble doing it would be nice for the team to know about it, as generating a certain amount of simple data with nothing applied as a background seems too light to me, maybe just needing refinement in the programming.

I believe in the potential of affinity and I use data merge where I work to make badges, coupons, numbered blocks, certificates and everything possible using variable data, but I use indesign most of the time. Take my attached file to test it out. Thanks

base.afpub Pasta de Trabalho1.xlsx

[Bruno Belo]

13 minutes ago, Bruno Belo said:

That's cool, thanks, nice to know there's this option. But I really wish we had a solution for data merging, because what I try to do is very simple, I could do it in MS Word, WPS Office, or Indesign too.

If this is something that AFP has trouble doing it would be nice for the team to know about it, as generating a certain amount of simple data with nothing applied as a background seems too light to me, maybe just needing refinement in the programming.

I believe in the potential of affinity and I use data merge where I work to make badges, coupons, numbered blocks, certificates and everything possible using variable data, but I use indesign most of the time. Take my attached file to test it out. Thanks

base.afpub 21.01 kB · 0 downloads Pasta de Trabalho1.xlsx 110.59 kB · 0 downloads

I did another test now, I lowered the amount to 5,000 numbers. By the way, I did two tests, one with 5,000 numbers and nothing in the background of the page. And another with 5,000 numbers and a pdf imported to the background of the page ( linked ), a scanned pdf with 5mb.

The 5,000 without the background applied generated normally and I was able to export. The 5,000 with the pdf in the background generated but the program closed when trying to export. Both had a consumption of 11 to 12gb of RAM in the generated file with 5,000 pages.

[Blake_S]

Something goes horribly wrong when AFP creates a merged file.

Just now a merged file with 412 pages, all having the same linked PDF from 2 master pages and a single text field on each page, has a 1420 MB (1.4 GB) size after being saved. I'm not able to export it, errors out every time.
This is stupid. It should be under 10 MB.
The linked PDF size is 1 MB and it contains 2 pages.
Besides the links to that PDF and text fields with personalised data, there is nothing else.

This suggests that AFP is doing something stupid, like creating a separate copy of a linked file for each page would be my guess.

Looks like its back to InDesign CS5 for good for me until this gets resolved.

___________________

Update - re-created exact same job in InDesign CS5.
Export took about 5 seconds, vs. AFP erroring out on export after like 3 or 4 minutes.

[Bruno Belo]

10 hours ago, Blake_S said:

Something goes horribly wrong when AFP creates a merged file.

Just now a merged file with 412 pages, all having the same linked PDF from 2 master pages and a single text field on each page, has a 1420 MB (1.4 GB) size after being saved. I'm not able to export it, errors out every time.
This is stupid. It should be under 10 MB.
The linked PDF size is 1 MB and it contains 2 pages.
Besides the links to that PDF and text fields with personalised data, there is nothing else.

This suggests that AFP is doing something stupid, like creating a separate copy of a linked file for each page would be my guess.

Looks like its back to InDesign CS5 for good for me until this gets resolved.

___________________

Update - re-created exact same job in InDesign CS5.
Export took about 5 seconds, vs. AFP erroring out on export after like 3 or 4 minutes.

Yes, i use data merge in long years, ms word, corel draw, indesign, planepress, ok some of programs are specific for this, but afp is in begin of this function.

Afp is a incredible software and i love it but data merge need more attention.

Out of data merge, i import 456 pdf pages linked from a pdf with less then 5mb. The use of ram reaches more than 20gb, i sucessfull export when i set the performance limit use of ram to afp to the max possible ( 65gb ) using 20gb ram of system and rest of virtual memory, only method to work.

 

[Blake_S]

So, will this be addressed? Or do we just give up on using the Data Merge function in Affinity?

Bug is still present in the latest version. Just now tried to data merge a simple ticket with only 600 numbers, which are placed into only one text field.
A 1.8 MB PDF is used as the background.
Realistically, this should take less than 5 seconds to merge.

After one minute and the drive running out of space because Affinity managed to write 40GB of data, I killed it via task manager and went back to InDesign CS5 to do the job instead...

As a comparison, InDesign CS5 took 4 seconds to export 600 records, 14 seconds for 3000 records, and didn't write any noticeable amount of data to the disk.