Publisher v2 doubling file size after each addition of a new data merged page : (

[Caleela]

Publisher 2.2.1  ~ Mac App Store ~ macOS 11.6.2 iMac ~ Acceleration ON ~

Hello...

I am re-making a digital planner for next year - last year was a manually edited of 700-ish pges. The file was 29Mb finished. I am using the same master pages, and this year I am data merging and making the merged pages as I go, and at only 10 pages, the file is 163Mb.

The key difference is last year was using Publisher V1, and not data merging with the same OS and hardware.

• I have a Master page with a Data Merge Layer (created with the Data merge layout tool with 4 columns x 31 rows). The merge layer has 4 fields mapped - each in a text frame. The data merge works sucessfully, and generates an untitled publisher file with the resultant page populated. This is saved as untitled.afpub.
• I add the merged page into my publisher file [Document | Add pages from file]. This is also successful.
• I add another master page to this (header, footer and left index). This works. This makes a month summary page.
• save...

3 Observations:

1. Importing the merged page also adds a duplicate of the master page [exact same name as the original] image attached. 
2. The file size doubles after adding a new merged page. The file started as a 5Mb file, and is now 163Mb with 9 pages. There are no images, backgrounds or heavily filled tables. Each of the 31x4 fields is turned into a group in the merged page.
3. The save and open times are getting exponentially longer as the file grows - making it impossible to keep adding pages via data merge. The plan is to add the 12 month pages manually with data merge (got 10 done), then a month of daily pages with one daily merge Master... It may take me until 2024 to get this done.

Add Page from file

Correctly merged page added to publisher document

Applied added Master page . This is OK.

 

:  )

File Size Increase :  (

[Dan C]

Hi @Caleela,

Thanks for your report!

I'd like to request a copy of your Publisher documents and Data Merge source file, so that I can further investigate this issue here.

Can you please upload:

• A copy of the .afpub file pre-merge with your Data Merge layout on the master page,
• The Data Merge source file
• The .afpub file this creates post-merge ("untitled.afpub")
• The .afpub file you have merged the Data Merge file into, where a duplicate Master page is created

https://www.dropbox.com/request/A6ZFirdRWsVpWcH5l2fJ

Once uploaded, please reply here to let me know - many thanks in advance

[Caleela]

Hi @Dan C,

Files uploaded, including a pdf of notes.

The upload took a few tries - there may be duplicates 😁

Cheers,
Caleela.

[Caleela]

@DanC 

I have tested this with a samll file and it appears that the size of the untitled.afpub is being added to the original publisher file when adding a (one) merged page from untitled.afpub. 

1. I had a 5.7Mb file with no merged pages (source.afpub)
2. Create a merged page untitled.afpub file (same size as the source.afpub) 5.7Mb.
3. Add one page from untitled.afpub to source.afpub & save. Source.afpub now 11Mb.
4. Delete merged page from source.afpub & save. still 11Mb.
5. re-add the merged page from untitled.afpub to source.afpub & save (16Mb) ==> Incrementing the value of the untitled.afpub file each time a page is added

The untitled.afpub file is also a duplicate of all the pages in the source file, not just the merged pages.

Cheers,
Caleela.

[Caleela]

Hi @Dan C,

I have updated to 2.2.0 and re-tested.
The Growing size issue still persists.

Cheers,
Caleela

[Dan C]

My sincerest apologies for the delayed response here as we have been very busy following the release of 2.2, though I should have updated this thread sooner.

I can confirm that this issue is currently under investigation by our team internally, as it does not appear to occur equally across all files, therefore I'm awaiting further information from our team before logging this issue with our developers.

I will be sure to respond here very shortly with our findings, and any suggestions we have to workaround this issue in the meantime.

Many thanks for your continued patience and understanding here

[Caleela]

@Dan C

Thanks for the update.  Looking forward to a solution.

[Dan C]

Thanks for your patience, using your files I've found the following:

• Opening 'Original MASTER.afpub' on my machine used over 22GB of RAM and failed to open after waiting 25mins with the 'loading' bar showing.
• Opening '01 trimmed Original MASTER.afpub' used similar resources and failed to load after the same amount of time.
  

I'm currently unsure why both of these files fail to load for me, but this may indicate an issue with the file itself, causing the bloated file size and the file size not dropping when deleting pages.

Therefore, I tried the following:

• Creating a new .afpub file from the 'No Place Like Home' (97mb) Sample file, then duplicating this document and merging one page from the duplicated file into the original resulted in a file size of 101mb - ie the file size was not doubled and instead increased an expected amount.
  Equally, the source file after merging only had 1 Master page with no duplicates as I'd expect.

On 9/18/2023 at 6:42 AM, Caleela said:

I have tested this with a samll file and it appears that the size of the untitled.afpub is being added to the original publisher file when adding a (one) merged page from untitled.afpub. 

1. I had a 5.7Mb file with no merged pages (source.afpub)
2. Create a merged page untitled.afpub file (same size as the source.afpub) 5.7Mb.
3. Add one page from untitled.afpub to source.afpub & save. Source.afpub now 11Mb.
4. Delete merged page from source.afpub & save. still 11Mb.
5. re-add the merged page from untitled.afpub to source.afpub & save (16Mb) ==> Incrementing the value of the untitled.afpub file each time a page is added

The untitled.afpub file is also a duplicate of all the pages in the source file, not just the merged pages.

Are you able to provide a copy of these files (either directly to this thread, or to the previous Dropbox link provided) so that I can test with these instead - as hopefully I should have no problem opening these smaller documents here and understanding why you might be seeing different results to me.

Can you also please include a screenshot of the 'Add Pages from File' dialog, so that I can repeat the same settings as you when merging one file with another?

Many thanks once again!

[Caleela]

Hi @Dan C

Yes, the file opening was lengthy (16GB Ram) - open and walk away for ages.

I eventually went back to v1 to see if the problem was there, and I had deleted the test v2 small files as they were useless to me.
The growing file size happens for v1 as well.
I have uploaded (via dropbox link) a small v1 edition with staged saves, the data spreadsheet, and a PDF with process and comments. All the data I need to make the document.
v1 AP files for support.zip 
Included in the upload is an unmerged AP v1 file I made last year with ~700 pages and only 24Mb finished. My small merged version is 33.5 MB with only 7 pages.

I'm assuming opening in v2 will get you to a small v2 file.
This is what I did originally to start the v2 merge file - opened the unmerged v1 file from last year, deleted all the pages and started configuring some master pages for merging.

My add pages from file config is typically add 1 page (of 1) from the saved merged data file. The screen grab is of an older add pages window, but the pdf uploaded describes what I am doing.

Cheers,
Caleela.

[Dan C]

I wish to provide an update regarding your reports @Caleela & my apologies for the further delay here, this issue is still ongoing in testing and I can confirm I've now been able to replicate the issue with your file size not changing after deleting multiple pages from your file etc - however I am still investigating the Data Merge and Add Pages From File issues.

This is unfortunately taking longer than I had hoped currently due to the 2.2 update being released recently and our technical support team being inundated with customer contacts at this time. We're doing our best to ensure that both new users/issues are addressed quickly, whilst trying not to allow previous reports to fall too far behind.

Due to the nature of this issue, the long delays between opening, editing and saving your document are also increasing the time it takes our team to fully investigate and report these issues to our developers.

I understand this issue is of importance to yourself, so I must apologise if this is seen as an excuse and I thank you once again for your patience and understanding here, it's certainly appreciated.

I will be sure to update this thread in due course regarding the other issues you have raised here

[Caleela]

Thanks for the update @Dan C.

[Dan C]

Thank you for your continued patience here - I've been doing further testing and investigating with your files and I have found the following:
__________

1. There is an issue with your 'Original MASTER.afpub" files provided in regards to the file size and opening times in the Affinity apps. Opening this file on my mac took a considerable amount of time (over 60 mins), used 33GB+ of RAM (memory) and read over 175GB worth of data from the disk. This is never expected with an Affinity document and indicates a larger issue with this specific file that I can't personally locate - therefore I have logged that specifically as a bug with our developers to be investigated further. I believe this is the same issue underlying issue causing the files inability to be 'reduced' in physical size - as I have deleted all pages & master pages from the document, essentially leaving a file with a singular blank page and the size of the saved .afpub file did not reduce. I have noted this within the development report for your file.
__________

2. Following the below steps with your V1 files (in both V1 & V2) I'm not seeing the additional masters being created - though this may be due to a misunderstanding in your workflow, as I do not mind admitting I've had a little trouble understanding the PDF file provided with these files. The steps I've followed are as such:

1. Open "V1 2024 BUJO Master 0 Pages.afpub"
2. Select "Master BUJO Links.xlsx" in Data Merge manager to relink
3. Set Sheet to "MergeFutureLog"
4. All Records, Merge Enabled, All Pages 
5. Generate from Data Merge
6. Save generated document
7. Return to "V1 2024 BUJO Master 0 Pages.afpub"
8. Add pages from file - select recently generated/saved doc
9. Add All Pages, Before Page 1, Discarding unused text styles
10. (❌) No new masters added to file?
11. Delete Page 2 (was Page 1)
12. Add new Page using "00 Year by Months LinP DM" master
13. Data Merge Manager
14. Set Sheet to "Merge Year Calendar", All Records, Merge Enabled, Page Range 2-2
15. Generate from Data Merge
16. Save generated document
17. Add Pages from file - select recently generated/saved doc
18. Add All Pages, After Page 2, Discarding unused text styles
19. (❌) No new masters added to file?

Am I misunderstanding the workflow you've described within the PDF provided for the V1 files? If so, could you point out amendments to the above steps that would replicate this duplicating masters issue? 
I did not continue past the first two Merge > Save > Add Pages from File attempts, as neither of these created duplicated masters and therefore I determined my workflow must be missing something.
__________

3. As I wasn't yet able to replicate the duplicating of masters, I don't believe I'm seeing the same file size increasing on 'Add Pages' that you've reported also. 
When adding the merged result document to the 'parent' .afpub file and saving it, the file size does increase slightly - but this is expected as new page(s) have been added to the document, from the merged files. Again I suspect that I'm not seeing the reported issue directly due to misunderstanding the workflow causing your duplicated masters.
__________

Hopefully with the correct steps for point 2 above, I'll be able to replicate both issues reported and log these with our developers also.
I look forward to hearing from you here

[Caleela]

Hi @Dan C,

Suspected PICNIC...

I started from a new afpub document (v1) and manually added my existing masters (C+P layers) from an old small file.

I split my Masters files into 2 files - with data merge and non-data merge masters.

1. The Data Merge file generates the pages to be merged.
   • and there are only 7 masters in this file:
     • 5 masters make a single page
     • 2 masters make a month's worth of pages (each month is merged to crreate a new merged month file)
2. The non-merged masters file is the basis of the final file. 
   • This file has loads (>20) of masters - headers, footers, month pages, etc
   • once I add the merged pages (for a month), I add the relevant headers, footers, etc.

I have sucessfully merged and added 157 pages (so far) and the file size is a tiny 2.9Mb

I think the following may have caused my reported issues:

1. my original (wing it) workflow of having all the masters in one file and merging with this repeatedly
2. on the fly editing of a master (either in original or merged file - they look the same) may have caused issue 2 when the paged were added from the merged file.

Cheers,
Caleela.