10 MB PDF consumes over 24 GB RAM - Is this normal?

[Intuos5]

Hello, I just opened this publicly available PDF file which is ~10 MB in size, yet consumes over 24 GB of memory at times. Illustrator couldn't handle removing the brown hatch, whereas Affinity Designer could. 😃 However, Illustrator only consumes slightly over 5 GB of ram upon opening for the same file, which dropped to 4 GB once Illustrator was recognised as process within the Task manager (Windows 10). Whereas Affinity had been over 20 GB while loading the file and over 24GB when actually working in the file (after having removed the brown hatch and having rotated the file by 90 degrees.

Values are CPU % | RAM usage | Disk write speed, it was at 96% so Designer had to use my SSD.

I would expect this to be a good file for the team to look at to see whether there are additional performance gains to be had from CAD exports.

I really have to praise the development team's efforts, because once open, the navigation is still buttery smooth compared to slow dinosaur that is Illustrator! I was septic at first seeing the 1.9 release video, but now I am impressed. 👍
I do hope this RAM issue could be sorted out.

06_3-5-6-2_PropVar_L3_2004A_P03.pdf

[Intuos5]

While working to reduce the file size, by getting rid of some of the hatches, I managed to reduce RAM usage back to 4 GB.
However, there's another issue at hand:

1. If I double click on text to select it within the nested groups,

2. If I then select any associated strokes and fill objects from the Leader, including multiple hatches for the dot mark within the layers studio

3. And delete these elements afterwards

4. The text tool is active as opposed to the Node Tool (A) or the Move Tool (V).

This is quite the contrary of what I would expect, since I have selected other objects and did not alter any text, apart from touching it when double clicking.

[Intuos5]

And this file fails to open at all, I have waited for about half an out before giving up on it. It was using 20 GB of RAM, owed to the hatch applied to the pavement, I suppose. In Illustrator it requires 2 GB of RAM, compared to about 20 GB in designer. I don't know why Designer had a write speed of ~300 MB/s that persisted for over 20 minutes, that shouldn't be the case.

E: Seeing as how hatches are probably the cause of the problem, would it be possible for Designer to recognise a recuring pattern and recreate this as a pattern/ hatch of its own? Although I see there is no such functionality at this point in time... Note that Autocad can recreate hatches, so from a technical perspective, it should be doable.

6_9_6_6_El_2002_vleMorgagni_4-1.pdf

[NotMyFault]

It's using RAM up to the given limit in Preferences, and the the swap file - possibly unlimited or until the disk is full.

Affinity is swapping to death between RAM and "PersonalBackstore". I have never seen Affinity Apps finish once started using the Backstore file.

I would Assume even a system with 1 TB RAM would be exhausted.

Fun part: during opening, you could hit the escape key to close the opening dialog and view (maybe edit) other already open documents. But the load continues in the background and cannot be stopped except killing Designer.

[NotMyFault]

Out of curiosity, a use Adobe Acrobat to export the PDF into EPS (~ 20 MB).

The EPS file can be opened in Photo / Designer, both using about 2 GB RAM.

It is shown as .afohoto because it opened by default in Photo, and i switched the to "Edit in Designer" - which took several minutes.

The afphoto file is again 10 MB.

Exporting again creates 2.5 MB PDF.

Opening PDF from Designer: takes several minutes to finish. No HDD activity, only 1 core utilized (from 16).

 

The structure of the file is really interesting.

There is a huge white area, 99% covered by other areas (grey street)

It is painted by >10.000 rectangles of 0.1pt height, all overlapping 50%.

The upper layers are build similar.

The PDF contains literally millions of lines each 0.t pt height, in multiple layers.

The cyan areas are combined of countless small rectangles.

[Intuos5]

@NotMyFault Yes, the file is a giant mess, an atrocity only Autocad could create with it's jagged and split up hatches.

Is there any advantage of using EPS over PDF? I personally never used the file format.

Since I cannot find anything about this in the documentation/ help files, wouldn't limiting the RAM usage increase read/write activity to the scratch disk? It didn't seem like Affinity uses excess RAM for additional performance, because when I opened the file in Illustrator in parallel, Designer did not release a portion of the RAM to be used for Illustrator, so they were both competing for RAM and neither could open the document really (considering I also used a couple of GB of RAM for small things, including the browser).

[NotMyFault]

There is another thread discussing the RAM settings and if they have any positive impact to performance of Affinity and/or parallel running Apps.

My view: The OS is far better in managing RAM and pagefile than almost all apps (maybe except database systems).

Reducing RAM in preferences will do much more harm than good:

If Affinity needs more than configured, it will do excessive disk IO (observed), even if it is not currently required by other apps at all. This degrades performance for all apps. The setting is static / not aware of RAM demand by other apps. It seems to be effective immediately.

Actually, someone should test using files proven to need several GB RAM when opened.

[Intuos5]

Thanks, that's exactly what I suspected.

12 minutes ago, NotMyFault said:

Actually, someone should test using files proven to need several GB RAM when opened.

I think the files I provided would be prime candidates 🙃

[Intuos5]

Interestingly, my RAM limit was set to 16 GB (the default half of the system size). But, it exceeded this limit by more than half! Also, increasing the RAM limit for the second file to 26GB and I am still running out of system memory, Designer used 26,5GB by then with only minor processes left running of less than 100mb each.

[stokerg]

On 1/5/2022 at 8:33 PM, Intuos5 said:

I would expect this to be a good file for the team to look at to see whether there are additional performance gains to be had from CAD exports.

 

With a PDF like this, which is 700k objects, a lot of them image layers, it's always going to be tough to handle this type of file.  When Affinity runs out of RAM, it will start writing to disk.  I opened this file on my M1 BookAir with 16GB in Designer and while it took over a 2 minutes to open, once opened i could work with it.  Panning was a little slow to update but it out preformed Ai for me, which promptly locked up while selecting an object and i had to use Force Quit.  This is something that the PDF creator should really tackle and find a better way to produce a PDF that doesn't contain so many unnecessary separate objects.  

 

[NotMyFault]

1 hour ago, stokerg said:

With a PDF like this, which is 700k objects, a lot of them image layers, it's always going to be tough to handle this type of file.  When Affinity runs out of RAM, it will start writing to disk.  I opened this file on my M1 BookAir with 16GB in Designer and while it took over a 2 minutes to open, once opened i could work with it.  Panning was a little slow to update but it out preformed Ai for me, which promptly locked up while selecting an object and i had to use Force Quit.  This is something that the PDF creator should really tackle and find a better way to produce a PDF that doesn't contain so many unnecessary separate objects.  

 

On Windows, the file does not open at all.

After consuming all allowed physical RAM (16 GB of 32 GB), Designer starts to excessively use the "PersonalBackstore.dat". My SDD delivers sustaining 300 MB/s write throughput, but naturally slows down when reaching the capacity limit.

After about 10 minutes, i had to kill the process when it reached 85 GB on disk (to avoid system disk full).

How much BackupStore did Designer finally use on Mac to successfully open the file?

How much sustained SDD write IO is used until that state?

After killing designer, out of curiosity I inspected the file with WSL / hexdump -C, and tried to zip it using 7zip. It is almost empty or contains repeated patterns, so the compressed file is about 250 Mb, compressed more than 1:100

[Intuos5]

@stokerg Yeah same here, Designer is snappier on the first file than Illustrator, but that's with a whopping 4x as much RAM allocated. Surely, the pixelated render preview makes a lot of difference, Illustrator's rendering engine is simply outdated compared to Designer's. Then again, I would tolerate Designer using twice as much RAM, but not this much more. Considering you only had 16 GB RAM to work with, there apparently is a major difference between M1 Macs and Windows that definitely needs to be looked into, because, for the second file, I also let Designer run for about half an hour before killing the task, like @NotMyFault. It didn't occur to me though to check the size of the file written to disk. I also had a sustained 300 MB write speed, prolonged speeds of 800 GB/s for a few minutes and spikes of 1 GB/s. So it's no wonder that the file becomes 85 GB, but that's for a 7 MB PDF, for which Illustrator used only 2,2 GB of RAM once opened, whereas Designer required 11+ times that AND 80+ GB disk space!!! Not to mention, Illustrator opened this file in what felt like less than two minutes, and I had already been able to finish editing the file by the time I would kill Designer...

Yes the file is a mess and it should be exported better than this, but it's an older file and its all I have to work with, so exporting the file again is not an option here. If anything I would have used Rhino to create and export the hatches, because it gives clean fills, unlike Autocad. It's not about the speed of Designer here, it's about its resource allocation, which is way more than its direct competitor.

If anyone is proficient with Inkscape I would be interested to see a comparison for resource usage and performance...

Don't get me wrong, I bought the Affinity suite because its performance appealed to me in smaller files, but its not yet up to this task. And I have high hopes Designer will eclipse Illustrator in all areas within the next few years.

[NotMyFault]

24 minutes ago, Intuos5 said:

its all I have to work with

Yo may try my trick to export as EPS (in Adobe Acrobat) and then import to Designer. Designer behaves ok then. You can even export to PDF, getting a cleaner and smaller file, which can be opened in about 2 minutes without excessive RAM usage.

I cannot check if there are relevant differences (for your planned editing) after those import/export steps. And as i don't know the legal status of that file, i don't want to upload my version here.

You might even be able to merge large parts of these nitty-gritty rectangles to far less curves, making the file better editable.

[Intuos5]

Thanks for the tips, will keep that in mind in the future! For now i already solved it by using Illustrator to edit the file, so job done.