C-drive filled 100% by Publsiher file; PersonaBackstore.dat

[mmmmmmmm]

Currently I am working on a family photo album that has up till now 60 of the 84 pages with images with an average of about 3-4 pictures per page. The image placement policy is set to prefered linked and the images are less then 2MB per piece. After opening the file the performance menu of Windows shows that the internal RAM is filled up to its maximum and a bit later the C:-drive start to goes 100%. This process take a up to 15-20 minutes for the current file and I this takes longer evey tiem I open the file. The C:-drive is 128 GB and normally has about 20Gb of free space. Affinity Publisher is installed on this drive while the Publisher file and the photos are stored on my 1TB D:-drive. Both are SSD's. I noticed that during editing the 100% for the C:-drive happens from time to time and this is related to the Recovey File Interval.

It writes to the file C:\Users\Gebruiker\AppData\Local\Temp\PersonaBackstore.dat. This file is now about 22,5 GB and is written at the moment I opened the file. When Publisher is closed the file will be deleted and there is again spaced free om the C:-drive. Now I have 17 MB of free space left.

Before I started this post I openened the file Publisher is still readind and writting to the already full disk. What is going on?

 

 

 

 

[v_kyr]

Since you seem to encounter space problems here, since your C drive is pretty full and Publisher will create in your case a very huge PersonaBackstore.dat backup file, you might want to create the "Temp" directory on another drive (D or what you have physically there, which then hopefully has much more free space left) and symlink that to C:\Users\Gebruiker\AppData\Local\. - So that C:\Users\Gebruiker\AppData\Local\Temp\ is symbolic link to a physical D:\Temp directory or the like.

To see how to deal with symlinks under Windows see ...

• The complete Guide to Creating Symbolic Links (aka Symlinks) on Windows
• How to take advantage of symbolic links in Windows 10
• ... and so on ...

 

[mmmmmmmm]

@v_kyr, thanks for the input. Instead of relocating the temp folder to the D:-drive I moved my lightroom catalog to there. This gave me about 30GB extra so there is about 55GB of free space om my C:-drive. It does help and after more then 20 minutes the file is loaded. I made photoalbums with InDesign and vendor specific software and encountered never a waiting time of +20 minutes to load a file. I know disk space is not the same as RAM space, however I am suprised by some numbers.

The total album is 84 pages and 57 are filled with images, text, etc.. On the remaining pages there is nothing placed. My laptop has 16GB of RAM and including the 30GB saved to the C:-drive the total load seems to be 46GB. it is probably not that straighforward but 57pages/48GB means more then 800 MB per page/1.6 GB per spread! My guts feeling does tell my that is quite a lot. What is causing this? Are these sizes normal? is ther anything that cn be done to prevent this?

With about 20 pages to go I think working with the file will become almost impossible.

[v_kyr]

Well that depends on several factors, namely how huge the 3-4 images per page really are and if the APublisher's linking option woud really link (?) and not instead just place a full copy of each image into the pages then. - Note that personally I don't use APublisher, since it doesn't offer what I need and IMHO is too buggy too, thus somebody else who uses it too can tell you probably better than I here.

[Pšenda]

9 hours ago, v_kyr said:

linking option woud really link (?)

Linking or Embedding only defines, how the inserted file is stored. That is, whether it is a direct part of the work file or not (part of the work file is only a link to inserted file). However, this has nothing to do with when the working file and its contents (including inserting files) are processed, displayed, or exported. It must always be completely loaded and processed - linking or embedding does not affect it in any way. If the working memory is insufficient (the inserted images are large and there are many of them), then they have to be swapped to disk - which is of course very inefficient.

 

On 12/20/2020 at 9:19 PM, mmmmmmmm said:

about 3-4 pictures per page

How many bytes (uncompressed/expanded) is that? I mean how many bytes in memory these files occupy, that is, approximately how many pixels x 4B, if you use RGB8, or x 8B if RGB16.

[v_kyr]

58 minutes ago, Pšenda said:

Linking or Embedding only defines, how the inserted file is stored. That is, whether it is a direct part of the work file or not (part of the work file is only a link to inserted file). However, this has nothing to do with when the working file and its contents (including inserting files) are processed, displayed, or exported. It must always be completely loaded and processed - linking or embedding does not affect it in any way. If the working memory is insufficient (the inserted images are large and there are many of them), then they have to be swapped to disk - which is of course very inefficient.

For Linking I would usually expect also a PersonaBackstore.dat backup file (?) then to contain just references to the images and not maybe all image files embedded here, but urging from the OP's told 22,5 GB file size I doubt that highly then. - The other things you told are normally just processed in RAM (working mem, as far as that is big enough) and not on an Affinity physical backup disk storage. If RAM is insufficent usually the OS performs swapping but probably not that PersonaBackstore.dat backup file!

So the question is more what services that PersonaBackstore.dat backup file handles, if cashing and recovery or other things too here.

[mmmmmmmm]

I did some back on the envelop math as described by @Pšenda it might makes things more clear. The document setting is RGB16 and for now I took an average of 15MP (camera's with different resolutions are used) per image, 4 images per page and 55 pages: 15e6 * 8 * 4 * 55 = 26.4GB. However, it still seems that this it not well dealt with within Publisher.

As I described earlier Indesign and other vendor specific software packages do much better job handeling this. When I worked with those packages I noticed things working somewhat slowly when the design of the photo album progressed, but not as dramatic as with Publisher. Never there was the need to wait more then 20 minutes to have a 25GB recreated every time I open the photo album file. If the files are linked why is there the need to store them in the PersonaBackstore.dat backup file ? What is in this file anyway and why does it need to be recreated every single time?

Since the files are linked for now I can lower the resolution (with an export from Lightroom) and work on. When I am done with the book I need to get to the right resolution (DPI) and things might get interesting. This does not mean I am happy with how it is handeled.

[Pšenda]

15 minutes ago, mmmmmmmm said:

26.4GB

Although we do not know exactly how Affinity stores its working image data, the calculation corresponds relatively well to the size of the PersonaBackstore.dat file, and it is also clear that this is a big portion of the data.

 

22 minutes ago, mmmmmmmm said:

If the files are linked why is there the need to store them in the PersonaBackstore.dat backup file ?

As I wrote earlier, whether the data is linked or embedded is not affected in any way by the fact that it must be loaded into working memory in order to be processed. However, this does not exclude, that it could be done more efficiently.

Backup file? Why? It is rather a swap/work file of the application for data that may not fit in RAM (each user/PC has a different size), and it is not necessary to use an OS swap file for them (each user/PC may have it set differently, or may have disabled).

 

32 minutes ago, mmmmmmmm said:

Since the files are linked for now I can lower the resolution

I think that the Draft/Preview mode would be suitable for processing large files, where only image previews are used.

[Pšenda]

36 minutes ago, mmmmmmmm said:

The document setting is RGB16

Try setting RGB8. It is still quite quality, but perhaps significantly less memory usage. 

[mmmmmmmm]

17 minutes ago, Pšenda said:

I think that the Draft/Preview mode would be suitable for processing large files, where only image previews are used.

Where can I find it? Searched on the internet and Publisher, but cannot find it. For now I followed this post:

 

16 minutes ago, Pšenda said:

Try setting RGB8. It is still quite quality, but perhaps significantly less memory usage. 

And then I export the document to PDF set it to 16?

[Pšenda]

11 minutes ago, mmmmmmmm said:

Where can I find it? Searched on the internet and Publisher, but cannot find it.

Yes, it was just a wish :-)

 

11 minutes ago, mmmmmmmm said:

And then I export the document to PDF set it to 16?

Yes, but really you need this super quality? PDF file will then be significantly larger.

 

Try read this:

https://www.diyphotography.net/8-bit-vs-16-bit-color-depth-use-matters/

[v_kyr]

18 hours ago, v_kyr said:

So the question is more what services that PersonaBackstore.dat backup file handles, if cashing and recovery or other things too here.

 

3 hours ago, Pšenda said:

Although we do not know exactly how Affinity stores its working image data, the calculation corresponds relatively well to the size of the PersonaBackstore.dat file, and it is also clear that this is a big portion of the data.

According to other forum sources (here, or here and here etc.) it's the way as Pšenda said, that PersonaBackstore.dat file seems to be used to cache/swap the apps working memory if memory isn't sufficent for assets like images and probably other related stuff. And in case of the OP it generates a huge file due to the amount of higher res images.

Thus if the C:\ disk hasn't that much free space left, one have to link (physically move) the working directory of the file to another drive with more storage space and just keep a reference on the initial C:\ drive. This way a huge growing PersonaBackstore.dat file on drive C:\ then shouldn't affect the drive size wise itself.

[mmmmmmmm]

On 12/22/2020 at 5:44 PM, Pšenda said:

Yes, it was just a wish 🙂

 

Yes, but really you need this super quality? PDF file will then be significantly larger.

 

Try read this:

https://www.diyphotography.net/8-bit-vs-16-bit-color-depth-use-matters/

I do not need that high quality, also because I do not perform any editing of images in Publisher. I might perform some slight (local) modifications with the Photo Persona, but that will be it.

For the current project I made an 8-bit versions, although the conversion took some time and disk space for now it seems to working very fine. Much fater loading of the file and my 16GB RAM seems in the end not to be filled completely. Thank you both for the help and I learned soem stuff again.

[Daniel the Schenz]

On 12/22/2020 at 8:43 PM, v_kyr said:

Thus if the C:\ disk hasn't that much free space left, one have to link (physically move) the working directory of the file to another drive with more storage space and just keep a reference on the initial C:\ drive. This way a huge growing PersonaBackstore.dat file on drive C:\ then shouldn't affect the drive size wise itself.

That's great info! Just, how do I do this? Note: I'm running it on macOS Big Sur.

Thanks anybody in advance!

[v_kyr]

32 minutes ago, Daniel the Schenz said:

That's great info! Just, how do I do this? Note: I'm running it on macOS Big Sur.

On MacOS aka Unix systems you create symbolic links (symlink), either via it's command line program from terminal.app or via some Finder extension or script. Search on Google after the topics "MacOS create symbolic links" etc. - Also look if something specific has maybe changed for Big Sur here in terms of system path access rights and the like.

• How to Create and Use Symbolic Links (aka Symlinks) on a Mac
• How to create a symbolic link on your MacOS
• ln, link
• Creating root-level directories and symbolic links on macOS Catalina
• ... and so on ...

 

[Old Bruce]

3 hours ago, Daniel the Schenz said:

That's great info! Just, how do I do this? Note: I'm running it on macOS Big Sur.

Thanks anybody in advance!

On Mac OS you can make an 'Alias' of your folder/directory on the external hard drive using the finder, it is different from a symbolic link but essentially functions the same way.

[v_kyr]

Yes using Aliases on MacOS is an alternative to symbolic links or hard links. - See also ...

• What Are Aliases, Symbolic Links, and Hard Links on a Mac?
• Wikipedia symbolic link (en)
• Wikipedia Alias (MacOS) (en)

[Daniel the Schenz]

Thank you, @v_kyr and @Old Bruce!

I created a symbolic link following the steps you have indicated. It was a bit tricky to locate the PersonaBackstore.dat file because it is so hidden, but after running a Spotlight search within "HD : private : var" I found it.

I just want to note, too, that there is a whole thread—in German, but whatever—on this same issue and the solution these lads (especially @PowerBug2.0, so thanks to you, too!) came up with was to prefer embedding files instead of linking them in the "File > Placement Policy" menu. That did increase their file size but for reasons unknown it did not exaggerate the PersonaBackstore.dat file so much.

Okay, thank you all for your help! I think this will work. If it doesn't, I will give you lads a little shout again!

DtS