Data Merge Takes up all my memory and HD

[Amandadianne]

Hey!

I recently discovered data merge and I'm putting it to good use. 

I want to be as efficient as possible and merge a lot of records but I'm hitting a wall.

Currently, I'm trying to merge 270 records with two fields that are images.

This eats up all my remaining memory and disk space. 

I have a 2017 Mac with 16 GB of memory and 250 GB of disk space running Big Sur. Normally about 7.5 GB of memory is available and 137 GB of disk space is available. 

When I do the data merge and hit generate, it might process 75% and then get stuck and my computer says there is no more memory. If I restart, it sometimes processes the data merge but I might get an error when exporting that there is no more disk space. And, sometimes I can do the data merge and exporting with no errors at all. 

I know I can do less records but I'd like to do as many as possible to be efficient, like I mentioned above. Is 270 records too much? Are my computer specs not good enough for that? I don't really know that much about computers but I could see why the app might take up the memory but I don't understand why it would take up disk space. 

 

[Old Bruce]

Make a new CSV or XSLX file with only two of the records and try that.

I have a Publisher document for generating Proof Sheets from my CR2 files, some of the folders have a couple of hundred images and to make a merged document means leaving it to run overnight. Granted I have 64 GB of RAM and about 400 GB spare space. But it should be doable, the reason I suggested using just two is to see if the image files are okay.

[walt.farrell]

32 minutes ago, Amandadianne said:

Is 270 records too much?

Pragmatically, if it doesn't work, then maybe it's too much and your machine can't handle 540 images in one document.

But check one thing. In your main Publisher document, check whether you have it set up to Embed or Link resources. You'll find that in File > Document Setup. The document generated by Data Merge will use the same setting as the main document. If the main document is set to Embed, then the merged document will also use Embedded, and you'll be embedding 540 images. That may be too many, but if you switch the main document to Link rather than Embedding, the merged document will use Linked, which will take less memory and may work.

 

[h_d]

I suspect it boils down to how big the source images are. You've got 540 of them. If their total size is greater than your 16Gb RAM (or indeed your disk size if you're embedding them) then I fear you're in trouble, because it's my impression that the data is all processed in RAM before being rendered to the individual pages. Your solution may be to reduce the size of all the original image files before the data merge. 

[Amandadianne]

I just checked and I have my settings as Link Resources. 

For my photo sizes, field 1 is 270 kb and 349 kb and field 2 ranges between 35 and 109 kb. 

I'm not sure if that is large or not but these files will be printed so I don't want to reduce the file size so I can keep the print quality. 

I guess I'll just do one batch at a time which is 130 images. 

[Old Bruce]

8 minutes ago, Amandadianne said:

For my photo sizes, field 1 is 270 kb and 349 kb and field 2 ranges between 35 and 109 kb. 

I think there is something wrong here, your machine should be able to do this. It could be the CSV file or, less likely, one or more of the images is corrupted. Are the images all on the computer's hard drive, not a networked or cloud?

[Amandadianne]

I have them on dropbox currently. I can try moving them to my computer but they will still be in iCloud since documents are stored in iCloud. 

BUT, I just ran it and it worked!!! I'm not sure why. It's frustrating when it works sometimes and doesn't work other times. 

I can also rebuild the csv on the next batch and see if that helps. 

[Joachim_L]

Perhaps it work different on Mac, but I had a project resulting in 2367 pages with one signature on it (in total 300 KB) and my AP document weighs 4 MB. You could link the files you want to insert to keep the document small.

[h_d]

7 hours ago, Amandadianne said:

I have them on dropbox

 

7 hours ago, Amandadianne said:

documents are stored in iCloud

Both of these can be risky. There have been a number of posts here from people who have had Affinity documents corrupted after working with them on remote storage platforms, and Serif's advice has always been to keep all working files on the local drive. Not sure if that's what was causing the original issue, but it's worth eliminating as a cause.

Keep backups... 😁