Document keeps getting corrupted Affinity Photo V2

[SandraS]

Hi there. I have posted before and even got a tech to look at the problem, but no joy, so hoping the community can help.

I scan images using an Epson graphic arts scanner. The images are 48 bit depth, uncompressed TIFs, typically about 1 GB off the scanner. 

I edit in Affinity and have done so through both Version 1 and 2. 

FIRST: my latest image, same parameters as described, has caused Affinity to choke twice, as I near completion in editing (which takes WEEKS). The file tries to load, and then I get a message that reads "failed to load document, document appears corrupted"

Here is the response I got:

"Hi Sandra,

I just wanted to let you know that I've downloaded your file. It also fails to open for me, so I've logged it with our developers to see if anything can be done. However I can't give any timeframe as to when this may be.

In reference to your other post and files getting larger, when this occurs try using File > Save As to create a new file. As this process should get rid of anything not being used and streamline the file."

So I started editing it again, and again, near completion, Affinity has choked. I have a Dell Precision laptop, 32 gigs RAM, 2.57Ghz processor. There is no reason that affinity should be choking on this. I have worked on scores of similar images. 

SECOND: (and alluded to in tech response above): the uncompressed TIFs come off the scanner at just under a GB. But as I work, the image bloats to 5 or 6 GB. I tried the advice above, of saving iterations under different names, but they are ALL bloated. Why does this happen? I thought maybe all the history was saved with every edit, but not so. I don't work in layers either - the document is flattened. I do need to routinely zoom to 400% for editing (painting in a true black back ground with the clone tool, using a WACOM tablet and stylus).

So. Any advice? Now Affinity got bought by Canva, let's hope it doesn't do a Boeing. Please advice why current version of Affinity chokes on large files, and how to stop the constant bloat in size. 

TIA.

 

 

[walt.farrell]

Where are you saving your files, Sandra? Ideally you would save them on a local disk drive, not managed by any cloud service (such as iCloud, Google Drive, OneDrive, etc.).

Also, if you're flattening everything, and working in pixel layers (not vectors), there's probably not much sense in saving as .afphoto files. Have you considered, rather than Save or Save As, using File > Export and creating TIFF files instead?

[SandraS]

Thanks, Walt! 

I save to a local disc. Have 4 TB on my laptop.  I figure that while I am editing, I should save as af files, until I am ready to export - but are you saying I could save each editing session as a TIFF? What will I lose by doing that?

First I run the TIFF from the scanner through a few Topaz filters. Then I use some of the affinity adjustments. Then I start using the clone tool to paint in the RGB000 background (which includes all the interstices) and repair damaged plants (tears, bruizes and so on). Sometimes I need to clone from other images. So with that work, I figured I needed to keep it "inhouse" as af file through the multiple editing sessions, but not sure why. What is the reason for the bloat?

I should add that when I am done, I do add layers for ten or so watermarks. Then export as a jpeg and upload to the shop.

 

 

[walt.farrell]

It's unclear what causes the bloat, if you're saving flatttened files (as you said you were) and using Save As under a new name. But without seeing one of your files, I really can't say more.

The files sizes will increase, generally, if you're saving Affinity format files (.afphoto, etc.) due to the way the files are designed, but the bloat is supposed to be removed by the Save As with new name.

Also, keeping the History will cause greatly larger files, but I think you said you're not saving the History? (That's an option in the Document menu if you're using Photo.) And if you're saving History, you may not be seeing bloat, but just the effects of saving all the data that the history requires.

[SandraS]

If there is a dropbox link you could send, I will send you the document. I WOULD save the history because in theory I could go back to a more recent point before the corruption, but that's in reality not poss because once the file is corrupted, I just can't access anything in it. 

At this point, I do some editing, save as an af with a diff name (same name with 01, 02, 03 etc) and every time I finish a session, I close and make sure I can open it again, and then delete older versions (EACH ONE in excess of 5 GB). This way I don't lose as much work - maybe several hours worth as opposed to weeks. But if Affinity is to be a real challenger to photoshop, which in many ways it is, I really like it, this is unacceptable.

Also, forgot to add, that the clone tool painting experiences serious lags. I have to wait, save the document, which can take a while, then start the painting again, until it slows again, then save, wait, repeat. Note that this is all new - the corruption, the lag - I have edited, as I said, scores of similar documents up to this point. (though the bloat was always an issue)

Computer recently put through a health check with firmware updates etc.

It is all very cumbersome. I seriously do NOT want to go back to photoshop, as it is far to pricey. 

Also, I don't typically flatten the file - I just work on the one pixel layer (until it's time for watermarking). I tried flattening. Made no difference.

I would really like to know what causes the bloat. 

Let me know if you want to see the actual doc.

Thanks.

 

[walt.farrell]

10 hours ago, SandraS said:

I WOULD save the history

The question is not whether you would do it, but whether you have been doing it. If "Save History with Document" is enabled (despite the warning when you set it) then it will cause greatly enlarged file sizes.

10 hours ago, SandraS said:

At this point, I do some editing, save as an af with a diff name (same name with 01, 02, 03 etc) and every time I finish a session, I close and make sure I can open it again,

A safer approach:

1. Save As with name-01. This leaves you editing name-01, but you have no idea whether the Save created a usable file on disk.
2. Immediately Save As with name-02. Now you are editing name-02, but again you don't know if the file on disk is usable.
3. Immediately Open name-01. If it opens successfully, keep editing in it and close the tab for name-02.
4. When it is time to Save As again, start with saving as name-02, and name-03. Then open name-02, and if successful keep working there and close the tab for name-03.

10 hours ago, SandraS said:

If there is a dropbox link you could send, I will send you the document.

If you have a cloud account you can save to (Dropbox, etc.) which allows file sharing feel free to upload a copy there and DM me a link. Or you could upload to WeTransfer and DM me a link to retrieve it.

 Or, at some point when Serif reads this thread they can provide one so they can examine your file. I'm just a user, and while I can share files to you I don't think any of the services I use would allow you to upload to my account.

[SandraS]

OK did all the save as and still bloated files. 

WAITING FOR A SERIF TECH PLEASE HELLO HELLO HELLO

[stokerg]

Hi @SandraS,

Your file from this thread is still logged with the Developers and it appears to be the same file that you show in your screenshot.

Do you have a version of that file that does open?  The file that's currently logged with the Developers closes on opening for me, so i can't look into what could be causing the issues.  Do you have a version of that file that does open?

[SandraS]

Hi Stokerg, yes, I do have a file that opens - need a dropbox link to upload. Depsite changing names, it still bloats to 5-6 gb. And for some reason I cannot see whether I have replied - had to sign in twice. Hopefully you don't get multiple responses. I can't see whether this has been submitted....

[stokerg]

Hi @SandraS,

I've set you up a Dropbox link here: https://www.dropbox.com/request/s0DPXm8UWkFdL1VxHhAc 

Let me know when you've uploaded the file and i'll take a look and see what i can find out  

[stokerg]

Hi @SandraS,

Thanks for uploading the file.

There are a couple of Saved Snapshots in that file.  If you call up the Snapshot panel (View>Snapshots) select each Snapshot and delete it.  If you then use File>Save As that will cut the size of the filesize down to 3.57GB

[SandraS]

Will do. Thanks. I will let you know if that reduced the number of times the file becomes corrupted, and the lag in editing.

[SandraS]

OK so I keep deleting snapshots. What are they good for? Is there someway to stop them? And the files are now 3-4GB rather than 5-6GB, but that still is rather huge (and space is a premium). And also, the lag hasn't gone away. Is this lag due to using a wacom tablet? But I have no other alternative, and have updated all drivers. Is this just normal? To have to make a small edit, and wait for the computer to complete it? When I stop for a while, something catches up, and the lag goes away, but then it starts up again. It's rather annoying...

[walt.farrell]

You may get one the first time you Open a file to begin a project. After that, you should only get one if you explicitly create it via the Snapshots panel.

Help: https://affinity.help/photo2/en-US.lproj/pages/DesignAids/snapshot.html

[SandraS]

Hey!

Something NEW has happened, which is also concerning!

For my latest image, which comes off the scanner at 1.5GB, after editing in Affinity Photo V2.4.2, the image has SHRUNK as an AFPHOTO to under 300MB???!!  (As opposed to bloating to 5-6X the size at 5-6 GB). This is GOOD because storage is becoming an issue with the aforementioned bloat size of AF files, but NOT SO GOOD in that the original scan is 1200 dpi so it can be expanded to large format printing. At 253 MB, is that still possible? 

I'm posting a composite image showing two editing sessions from two different scans - one made 15 April, one made 20 April, before and after editing. I don't recall an update to Affinity Photo in between. But something has changed - now the images are seriously compressed. I tried exporting as a TIFF to see if the larger size would return; it came back as 68MB????!!! Tiny.

And there is still a lot of lag in editing with the clone tool. Make an edit, wait for the computer to catch up, start again. Possibly because I scan at 48bit colour depth? I frequently have to zoom to 300% to edit.

Explanation would be most welcome!

 

[walt.farrell]

The on-disk size is important, but for figuring out what's going on it's more important to look at the pixel dimensions and color format & bit depth of the original and current files, I think.

[SandraS]

OK, here is what I could find - as I said, scanning is at 1200 dpi, colour depth is 48 bit, and attached, some settings in Affinity, of an unedited TIFF. The colour profile is Adobe RGB-IEC61966-2.1, I cannot figure out where to find out what the pixel dimensions are....the size of the individual pixels? Or the size of the image in pixels (14639 X 20640 as TIFF right off the scanner). 

Nothing has changed - in terms of what I do - between 15 and 20 April, when the two different images in the last post were generated. So I cannot explain the difference in AFPHOTO size after editing. From 15 April, file exploded in size. From 20 April, file shrunk in size.

I am wondering if I could change some of the performance parameters to help with the editing lag - not sure how to fiddle with that without affecting the image. 

Help welcome. 

 

EDITED TO ADD: when I take the 20 APRIL image (1.69 GB off the scanner, 1200 dpi, 14639 X 20640 pixels, 48 Bit depth) and export as TIFF (just to see if Affinity does anything to it), it shrinks down to 410 MB, and now only 8 bit. 

[walt.farrell]

Thanks.

What does Affinity Photo say about the dimensions and color format of your .afphoto file?

E.g., when you Open the .afphoto file, the information that is shown immediately in the Context Toolbar?

[SandraS]

Hi Walt - see attached - the edited is of course trimmed down, but I don't think that can explain the difference of 1.68 GB pre-editing and 253 MB post-editing. I wonder if somehow Affinity has decided (all on its own lol) to dramatically reduce the bit depth of the affinity file? I scan at 48 colour bit depth because I always felt that it gave me more editing headroom, but I note that when I open the scanner TIFF file in affinity and export as a TIFF, just to see what happens, Affinity automatically converts it to 8bit.

So my habit has been to keep as much info in the AFPHOTO as possible (these tulip images took only a few days to edit, whereas the larger images take weeks) and only reduce to 8 bit and compression for printing at the end - I export jpegs that are sized appropriately - sometimes just for cards, sometimes almost wall sized - and they are all 8 bit which is fine for the printer. The goal is to keep as much info in the stored edited version as possible. Previously, or prior to the 15 April edit, that meant that the AFPHOTO got huge (though I could carve off a certain amount by deleting snapshots as per previous advice). But now, AFPHOTO files seem to ditch information right at the start, (or compress dramatically? To explain the dramatic reduction in file size). This could be problematic for larger format prints I may need to make.

[walt.farrell]

1 minute ago, SandraS said:

but I note that when I open the scanner TIFF file in affinity and export as a TIFF, just to see what happens, Affinity automatically converts it to 8bit.

The default may be 8-bit, but you can choose another Export Preset, or choose another Pixel Format down in the Advanced section of the Export options:

 

As for the file sizes:

• The unedited TIFF may be compressed, or not. If not compressed it may be much larger than if it's compressed, but if it is compressed then one needs to also consider the kind of compression that is used. Affinity supports Zip and LZW compression. Zip is (I think) preferred for 16-bit images as I've heard that LZW does a poor job of compressing them. But I have not played with that myself.
  
  Have you checked to see whether your scanner is producing compressed or uncompressed TIFF files?
  
  In any case, just looking at the data, an RGBA/16 file will have 64 bits of data per pixel (4 * 16), and with dimensions of 14,639 x 20,640 px that will take 2,417,191,680 bytes of storage, or an uncompressed size of 2,305.2 MiB plus some extra for overhead & control information. So if you're seeing 1.6 GiB then you probably have a TIFF file that is compressed in some way.
   
• Your edited .afphoto file would need 64 bits per pixel, still, and with its dimensions of 11,677 x 19,335 px that should require at least 1,722.5 MiB. That makes your .afphoto file size of 253 MiB suspicious, except that much of it is black and I don't have any information on how well that might compress when being written to disk as a TIFF file. It may compress significantly.

If it's a file you can share here, I'd be happy to look at it and see if I can figure anything out. Or perhaps someone else will be able to.

[Old Bruce]

1 hour ago, SandraS said:

I cannot figure out where to find out what the pixel dimensions are....the size of the individual pixels?

The size of the pixels are ... well, one pixel x one pixel. If your image is 14639 w X 20640 h pixels and your DPI is 1200 then the image is (14639/1200) inches wide. If your image is 14639 w X 20640 h pixels and your DPI is 300 then the image is (14639/300) inches wide. In the first case the pixel size is 1/1200 inches x 1/1200 inches, in the second case the pixel size is 1/300 inches x 1/300 inches. 

I leave the arithmetic as an exercise for the reader.

[SandraS]

Thanks Old Bruce - useful calculations to have.

Walt - pretty sure the TIFF off the scanner is compressed, though only from the evidence you indicate given size - I find no info in manuals or online to confirm. 

STRANGELY (strangeness continues) I opened the original 20 April off-the-scanner TIFF in affinity (1.68GB) saved as an AFPHOTO and NOW it's bloated to 4+GB (instead of shrunk). Opened it again, looked for snapshots, weren't any. Did I do something early on in the 20 April File to reduce it in someway, without knowing? Problematic, because I am not sure how this much smaller finished file will blow up for printing as required. I guess I can start over (annoying).

SIGH. For what it's worth, here is my routine.

I open the TIFF (1200dpi 48bit colourdepth, typicallyt 1-2GB) from Epson Graphic arts 12000XL scanner. Here's what I do with the image.

1. Crop it close to the object (remove messy black background - image is scanned under a black velvet hood, but it's full of schmutz)

2. rasterize and trim cropped image, save as AF PHOTO (colour profile as described earlier somewhere). This is when the image typically quadruples or more in size.

3. Open image again, delete snapshots if any.  Flatten image, save again. Image typically still 4+GB (bloated) but if snapshots removed, can go down to 2-3GB. (Except for the 20 Ap edit which shrunk right away - this is the mystery).

4. Open image again, expand (resize) canvas and fill with RGB 000, flatten.

5. Start clone tool painting in true black with a wacom tablet and stylus. Object (tulips for ex), goes through a bunch of TOPAZ filters, also some colour balance in Afffinity. I keep saving as AFPHOTO - keep checking file isn't corrupted, transfer uncorrupted file to separate drive in case of corruption - this is a pain. But anyway the editing continues for weeks or days - the lag issue I might be able to experiment with performance parameters, not sure.

6. When editing is done, add layer with watermark and save as AFphoto. File is typically 3-5GB.

7. When ready to upload for commercial printing, export as Jpeg with dimensions as per order (card, poster etc).

Main issues I have, if I am to continue with affinity photo:

1. unpredictability for corruption of files - has happened twice in the last 8 months (and I've been using Affinity for 2? 3? years now - right from the start). To protect myself from this issue, I have to continually save my working AF file to a separate drive, after testing that each iteration isn't corrupted. I used to back up the original file off the scanner, then save a couple of editing iterations, but not every single one.

2. unpredictability for file size collapse or bloat. Something in between would be nice.

3. lag time in clone tool painting. Need to know how to fix that. Change performance parameters? Attached a shot of the current settings. Don't know what I should change to make clone tool painting (with a wacom tablet) smoother, without affecting file size/quality.

Thanks for puzzling through this with me. I really don't want to go back to photoshop.

 

 

[walt.farrell]

Thanks for the workflow details.

I would remove the Snapshot in step 2, before Saving.

Then, in later steps where you refer to Saving, does it help if you use Save As to a new name, instead?

As for the April 20 edit which shrank earlier than you expected: Save is supposed to compress out wasted space when there is something like >= 20% that could be saved. Otherwise Save As to a new name is needed.

 

[SandraS]

Hi Walt - yes I was advised to save as a diff name, which didn't work. SO far, only the checking for snapshot and remove works to reduce file. 

Will keep buggering on.