Huge file size after RAW developing with embedded Raw-Layer

[Guenterm]

Hello, after opening a 40 MB Raw file from my olympus and choosing Output-Raw-Layer(embedded) I developed the file with no further changes in the develop modul and saved it as afphoto file. The file size is now 700MB. 

i would understand if it will grow to 100MB because of originalfilesize+ copy of originalfilesize + someinternalthings ...but 15 times of original.

And opening the same file in Version 1 is a little bit faster than in version2 ?

All the other stuff I've had time to explore looks very well 👍

[Thane5]

Same issue here: a Compressed CR3 raw file (by Canon) that is only 15MB by itself, ends up beeing 178MB when saved as a embedded raw file in APhoto. Not as much of a crazy file size, but i'll probably rarely ever use embedded raw, especially when the linked one exists.

[walt.farrell]

What size is your .afphoto file, @Guenterm, in V1?

 

[Guenterm]

Dear Walt,

because the first orf was from a OM-1 which AP1 could not read I've made a new test with an orf from EM-5MII

 

and here from an other file...30 times bigger

best regards

[sbp]

On 11/10/2022 at 4:02 PM, walt.farrell said:

What size is your .afphoto file, @Guenterm, in V1?

 

How can we embed a RAW file in Photo v1?

[walt.farrell]

2 hours ago, sbp said:

How can we embed a RAW file in Photo v1?

You can't. But that is not needed to answer my question.

[NotMyFault]

On 11/10/2022 at 8:04 AM, Guenterm said:

Hello, after opening a 40 MB Raw file from my olympus and choosing Output-Raw-Layer(embedded) I developed the file with no further changes in the develop modul and saved it as afphoto file. The file size is now 700MB. 

i would understand if it will grow to 100MB because of originalfilesize+ copy of originalfilesize + someinternalthings ...but 15 times of original.

And opening the same file in Version 1 is a little bit faster than in version2 ?

All the other stuff I've had time to explore looks very well 👍

This is normal:

• depending on chosen color format (RGB/8, /16, /32) Affinity needs 2 or 4 times the space for additional bit depth
• all layer includes an alpha channel, plus 33% size
• Affinity Photo adds an „initial snapshot“, doubling the size
• Affinity‘s native format is optimized for speed, not for size
• RAW files actually use far less sensor pixels than advertised image pixels. The e.g. 24 MP are kind of upscaled/interpolated, using individual R G B sensor pixels, to create several image pixel with 3 color channels. That is one reason why digital camera images look always a bit blurry before applying sharpening.
• The only exceptions are Sigma foveon cameras, and Leica (or other brands) monochrom cameras. The Foveon sensor has the 3 color sensors stacked above each other, avoiding the blurriness of all other sensors with bayer pattern. 
  https://en.wikipedia.org/wiki/Demosaicing
• https://en.wikipedia.org/wiki/Foveon_X3_sensor
• When you edit the file, Affinity may use additional space to store your edits, and wont reclaim locations with old (garbage) data immediately. 
  Garbage collection can start later, when certain thresholds are reached.
• save copy is one method which could reduce file size. 

[Chris B]

21 minutes ago, NotMyFault said:

This is normal:

• depending on chosen color format (RGB/8, /16, /32) Affinity needs 2 or 4 times the space for additional bit depth
• all layer includes an alpha channel, plus 33% size
• Affinity Photo adds an „initial snapshot“, doubling the size
• Affinity‘s native format is optimized for speed, not for size
• RAW files actually use far less sensor pixels than advertised image pixels. The e.g. 24 MP are kind of upscaled/interpolated, using individual R G B sensor pixels, to create several image pixel with 3 color channels. That is one reason why digital camera images look always a bit blurry before applying sharpening.
• The only exceptions are Sigma foveon cameras, and Leica (or other brands) monochrom cameras. The Foveon sensor has the 3 color sensors stacked above each other, avoiding the blurriness of all other sensors with bayer pattern. 
  https://en.wikipedia.org/wiki/Demosaicing
• https://en.wikipedia.org/wiki/Foveon_X3_sensor
• When you edit the file, Affinity may use additional space to store your edits, and wont reclaim locations with old (garbage) data immediately. 
  Garbage collection can start later, when certain thresholds are reached.
• save copy is one method which could reduce file size. 

Essentially, this.

[sbp]

6 hours ago, walt.farrell said:

You can't. But that is not needed to answer my question.

The problem was with an embedded file? How can someone answer your question?

[walt.farrell]

42 minutes ago, sbp said:

The problem was with an embedded file? How can someone answer your question?

The user I asked (the OP of this topic) answered the question. And @NotMyFault described the reason for what the OP asked about. Probably better than I would have, but it's where I was headed, too.

[Corgi]

Sorry if I'm being dense, but I'm not really following this (and I presently cannot run experiments as I'm waiting for the MSI installer).

If I understand correctly, embedding a raw file can cause the resulting afphoto file to be far, far larger than the sum of the original raw file plus the size of what the afphoto file would be if it was merely linked instead. So in other words:

R = Raw file size
L = afphoto file size if raw file is linked
E = afphoto file size if raw file is embedded

and the situation is that E >> R + L

If that's the situation, I struggle to understand why this would be, despite the explanation by @NotMyFault. Why can't the original raw file be copied byte-for-byte into the afphoto file, and then treated the same way that a linked file is treated? Why does it the raw file need to be modified at all when it's embedded?

At first I presumed that to get suitable performance, the raw file needed to be pre-processed for performance reasons, thereby expanding its size. But if the performance of linked raw files is fine, then performance wouldn't be a showstopper for embedding a byte-for-byte copy.

And even if performance is a consideration, there could be four Develop options instead of three:

• Pixel
• Linked to Raw
• Embedded Raw, filesize optimized
• Embedded Raw, performance optimized

What am I missing?

[Corgi]

OK, well my question stands.

I just fired up an old unused PC so I could perform a Windows installation and test this out. I checked it out with two different Raw files (.CR2 and .NEF). I only did a few tweaks in the Develop Persona, selected Linked or Embedded, and then hit Develop and immediately saved. Here were the results (rounded):

Original NEF file size: 9MB
If linked, 1MB
If embedded, 73MB

Original CR2 file size: 13MB
If linked, 1MB
If embedded, 70MB

I also timed re-opening these files, and there was no appreciable difference among the four different saved files.

So my question remains: why on earth isn't the embedded file size approximately equal to the size of the original Raw file? The explanations so far aren't very convincing.

[NotMyFault]

As long as you not really edit the file, linked does not copy any pixel content from the RAW files. But just use merge visible, now you get the same file size as in the embedded case.

It seems linked is doing what the name says. Opening a file in linked mode is much slower vs. opening embedded, as it has to re.develop the file again which takes some seconds.

Embedded seems to store a already developed pixel layer in addition to the raw file, and this makes the size difference.

[Corgi]

9 minutes ago, NotMyFault said:

As long as you not really edit the file, linked does not copy any pixel content from the RAW files. But just use merge visible, now you get the same file size as in the embedded case.

It seems linked is doing what the name says. Opening a file in linked mode is much slower vs. opening embedded, as it has to re.develop the file again which takes some seconds.

Hmmm, this isn't what I'm seeing at all.

I added a pixel layer with a gradient (Overlay) after developing a linked file, and an embedded file. Then I did Merge Visible on each, and saved them. So that yielded files with three layers each: the RAW file, the gradient, and the merged.

The linked file grew from 1MB to 91MB.
The embedded file grew from 70MB to 205MB.
They both took the same amount of time to open.

Just looking at my experiments alone, my preliminary conclusion (for my purposes) would be that I don't like this tradeoff. I'd rather that the Embedded option just have an identical copy of the RAW file stored along with it, instead of expanding/exploding it. Even if it takes a bit longer to open, I'd rather save 100MB on each file.

 

[walt.farrell]

13 hours ago, Corgi said:

Just looking at my experiments alone, my preliminary conclusion (for my purposes) would be that I don't like this tradeoff. I'd rather that the Embedded option just have an identical copy of the RAW file stored along with it, instead of expanding/exploding it.

But it can't just have that. The RAW data does not contain an image, and you wouldn't be able to see anything on your screen.

It must be converted into something like an Image layer, in addition to having the RAW data (in some form) embedded. I do not know if the RAW part of it is truly just the binary RAW data, or if's been preprocessed (demosiac, etc.) to at least give pixel data. That would be bigger, but would save time when going back into the Develop Persona.

Edited November 22, 2022 by walt.farrell
Typo

[Corgi]

1 hour ago, walt.farrell said:

But it can't just have that. The RAW data does not contain an image, and you wouldn't be able to see anything on your screen.

It must be converted into something like an Image layer, in addition to having the RAW data (in some form) embedded. I do not know if the RAW part of it is truly just the binary RAW data, or if's been preprocessed (demoniac, etc.) to at least give pixel data. That would be bigger, but would save time when going back into the Develop Persona.

No, I realize it can't be just the contents of the RAW file. By just, I meant simply, as in, simply store the RAW file in its original form within the embedded afphoto file.

I presume that you're correct in that it's pre-processed, which makes the file larger. That's what I'm questioning.

I cannot think of a reason why it wouldn't be possible for there to be an option for the embedded RAW file to be stored in the afphoto file in its original form, unprocessed. That way, the size of an embedded RAW file would be approximately equal to the size of the original RAW file plus an equivalent linked RAW afphoto file.

Yes, it might save some time by having embedded RAWs pre-processed (it didn't in my quick tests), but I would rather keep my file sizes down than save a few seconds each time I opened up the file. If there's only one way to store embedded RAW files, my vote would be to optimize for file size, not speed. But why not offer both options?

[walt.farrell]

Thanks for that clarification.

[carl123]

15 hours ago, Corgi said:

OK, well my question stands.

I just fired up an old unused PC so I could perform a Windows installation and test this out. I checked it out with two different Raw files (.CR2 and .NEF). I only did a few tweaks in the Develop Persona, selected Linked or Embedded, and then hit Develop and immediately saved. Here were the results (rounded):

Original NEF file size: 9MB
If linked, 1MB
If embedded, 73MB

Original CR2 file size: 13MB
If linked, 1MB
If embedded, 70MB

I also timed re-opening these files, and there was no appreciable difference among the four different saved files.

So my question remains: why on earth isn't the embedded file size approximately equal to the size of the original Raw file? The explanations so far aren't very convincing.

Try with larger RAW files

I just tried a 100MB CR2 file

The AFPhoto file with the Embedded RAW file opened "instantly" as did the AFphoto file with the Linked RAW file

BUT (and it's a big but) the AFPhoto file with the linked RAW file only showed me a low-res (placeholder) image. Only 15 seconds later did this image update to full-res 

Is this because the linked unprocessed RAW files had to be processed again? (Note: The approximate time to develop that RAW image is 12 seconds)

Also, the thumbnail in the Layers panel did not appear until the full-res image appeared 

[Corgi]

26 minutes ago, carl123 said:

Try with larger RAW files

I just tried with a 60MB CR2 file, and can confirm your findings. Both files opened nearly instantly, with that big caveat. The embedded version was fully ready-to-go, but with the linked version, I had only a low-res image to work with, and unpopulated layer thumbnails. It took 30 seconds for the linked version to be fully populated (this is on kinda old hardware, an i7-2600 with a conventional hard drive).

On the plus side, I could still do editing on the linked version immediately even when only the low-res version was showing. I could add adjustment layers, live fills, a pixel layer, etc., and was able to see the results in real-time, albeit still at low-res until the loading process really completed. I wouldn't want to try pixel-level editing under the low-res conditions, however!

So the RAW file was 62MB, the linked afphoto file was 1MB, and the embedded afphoto file was 385MB. So in essence, the tradeoff for saving 30 seconds each time I opened the file is an extra 320MB of disk space.

Look, I can understand why some people would prefer the better performance of the current implementation of embedded RAW.  They open files frequently, and/or aren't so worried about disk space. 

But I feel a bit gaslit by @NotMyFault and @Chris B for saying that the large file size of embedded RAW is "normal" behavior. It's normal in the sense that it's expected behavior given the way they chose to implement it. But it's not the only way to implement it!  I'm concerned about disk space. It's wear and tear on my SSD, it takes longer to transfer and back up, it takes more cloud backup space. 

Interesting, if you look at the tutorial by @James Ritson you will notice that he doesn't even mention the difference in performance. He only mentions file size as the tradeoff for the convenience of having an embedded RAW file.

It's pretty clear at this point what's going on. I'll just post in the Feedback section my requests for:

• An Embedded RAW option that optimizes for file size rather than speed
• That the Linked RAW method use (or have an option for) a higher resolution "low-res" image. Right now, it's really low-res!

[carl123]

43 minutes ago, Corgi said:

An Embedded RAW option that optimizes for file size rather than speed

So basically, you want the unprocessed RAW file stored in the AFPhoto document (to save on disk space) and every time you open that document the RAW file is reprocessed, just like a linked RAW file needs to be

45 minutes ago, Corgi said:

That the Linked RAW method use (or have an option for) a higher resolution "low-res" image. Right now, it's really low-res!

The time it takes for the low-res image to change to hi-res will be based on the computing power of your PC/Mac. Most professionals dealing with RAW files all the time will probably already have very high spec PC/Macs, so may not even notice the low-res image or it may only appear for a second or two for some users.

 

49 minutes ago, Corgi said:

I'm concerned about disk space. It's wear and tear on my SSD, it takes longer to transfer and back up, it takes more cloud backup space. 

I don't think you will get any better than the Linked file option. Modern backup software only ever has to backup that linked file once (because it never changes) but every time you modify a AFPhoto file with an embedded (unprocessed or processed) RAW file in it, it will still need to do a much larger backup than a AFPhoto file which only has a linked RAW file

[Corgi]

1 minute ago, carl123 said:

So basically, you want the unprocessed RAW file stored in the AFPhoto document (to save on disk space) and every time you open that document the RAW file is reprocessed, just like a linked RAW file needs to be

...

I don't think you will get any better than the Linked file option. Modern backup software only ever has to backup that linked file once (because it never changes) but every time you modify a AFPhoto file with an embedded (unprocessed or processed) RAW file in it, it will still need to do a much larger backup than a AFPhoto file which only has a linked RAW file

Yes, to your first statement.

And I do agree with your last statement as well. I'll probably stick with linked files unless Serif provides an embedded solution optimized for disk space, which would be a happy medium for me. I do like the encapsulation of everything into one file.

At times like these, I think back to my first PC purchase in the 1980's, when I paid extra money to upgrade the hard drive from 10MB to 20MB. It cost a lot, but that drive was so roomy!

[NotMyFault]

3 hours ago, Corgi said:

But I feel a bit gaslit by @NotMyFault and @Chris B for saying that the large file size of embedded RAW is "normal" behavior.

This maybe is a minor  cross-talking.

I overlooked the „embedded“ from the title (when reading long thread the title often scrolls out of view). What i explained was the size difference between RAW files and pixel layers. I did not go into the impact of embedding a RAW into native afphoto files. The raw files are almost neglectable in size compared to pixel layers.

from 4 theoretical options Affinity implement all except one (that you would like to see), for good reasons.

1. V1: no relation to RAW file after finish develop. Only pixel layer (plus snapshot)
2. V2 linked: does not create pixel layer unless it is needed for later edits, low res preview, long start time until re-developed.
3. V2 embedded: embeds RAW, but always stores develop result as pixel layer to provide immediate full resolution rendering
4. not available: embed RAW without pixel layer. 

The think is: If you won‘t do any edits in Photo at all, why using Photo? It is a Photo editor, not a DAM.

 

[Corgi]

2 minutes ago, NotMyFault said:

...

The think is: If you won‘t do any edits in Photo at all, why using Photo? It is a Photo editor, not a DAM.

I think you've jumped to the conclusion that if you start editing a linked RAW file that the size difference versus embedded RAW disappears. In my experiment, it didn't. In fact the size difference got larger for some reason. Here are the numbers:

RAW file size: 13MB     
    unedited afphoto file:  1MB (if linked), 70MB (if embedded)   Result: ~56MB extra space consumed
    edited afphoto file (pixel layer (gradient) plus Merge Visible): 92MB (if linked), 205MB (if embedded)   Result: ~100MB extra space consumed

It's probably true that when you open a file with a linked RAW file that the program re-develops it into a pixel layer, as you said, but the space it consumes is either in RAM or on disk as temporary storage such that when you save and close the file, that expanded information is discarded and not saved with the linked afphoto file.

And, I should also mention that you can do plenty of photo editing without adding a pixel layer. I added a Mesh Warp filter and Curves adjustment to the RAW file above, and the linked file size was still about 1MB.