Alpha Channel Packing / editing Alpha Channel as regular color channel

[NotMyFault]

It's 2023 and we still get frequent request to "edit the alpha channel as normal color channel", and export / import files where the alpha channel is used for e.g. depth information (z-axis) or something else / not for alpha.

I support these feature requests to allow easier editing of alfa channels using all brushes, filters, adjustments etc.

Never the less, it is perfectly possible to do this even today with Affinity Photo, Designer both in V1 and V2.

Opening a file

The key is to separate alpha channel as greyscale layer once you open such a file, and fill the alpha channel to recover RGB pixel values where alpha was 0.

1. Open a TGA, PNG, TIFF file which contains exactly one RGBA layer. Assuming the layer name is "background" 
2. Use channels panel, click on background alpha, and "create greyscale layer"
3. Use channels panel, click on background alpha, and "fill alpha"
4. Rename the greyscale layer to "alpha"

You can now edit the alpha layer with all tools.

Notes:

• keep the alpha channel in background always at 100% (if not: channels panel->fill).
• keep the alpha layer in greyscale (all pixel should have identical values for R and G and B channel for one pixel, do not colorise)
• (if not->HSL adjustment, set SAT to 0, merge down)

In case you want to see the alpha channel acting as traditional alpha channel:

1. Create a channel mixer adjustment
2. Choose alpha channel (output)
3. Set "red" input to 100% (all color channels should have identical values)
4. set „alpha“ input to 0
5. Nest channel mixer layer in masking position to alpha layer
6. Nest alpha layer to background layer

You can continue to edit both layers individually, but you see a preview of the document using alpha layer as alpha channel.

Exporting the file

To export the file:

• unnest the alpha channel layer from background layer, move it to top
• rasterise to mask
• The file should have exactly 2 layers: alpha on top (mask layer), background on bottom (RGB channels, alpha filled 100%)
• Export as TGA, PNG, TIFF etc

Closing notes

• Gamma is not handled correctly for alpha channel when simply mapping from alpha to color channel or vice versa. RGB channels need gamma correction (normally 2.2), whereas alpha stays linear for rendering. If the difference is critical, you can correct the gamma with help of a procedural texture filter.
• I have tested the import and export steps repeatedly with TGA files in Photo V1.10, V2.04, V2.1
• This method was not invented by me, I simply wrote this tutorial to have a URL which I can use if someone raises questions or feature request concerning this topic.

Edge case (added 24.04.2024)

• It is not possible to correctly export with empty mask (alpha zero for all pixels).

Edited Tuesday at 07:46 PM by NotMyFault
Added edge case alpha all zero

[keena]

It's 2023. Just follow these simple 20 step by step points and notes to edit the alpha channel. It's so easy!

At least it might work. Thank you for the efforts!

[henryanthony]

@NotMyFault Thanks for these instructions. I will save them to my special instructions file for use in the future. 

[RC-1290]

Describing this as perfectly possible feels a bit like presenting a command line interface and saying it's perfectly possible to create any texture you want.

It might be possible, but not without a PHD thesis on the subject.

Don't get me wrong: I appreciate that you put effort into documenting the behavior of this software. But this does not suddenly make it a feasible workflow.

All I wanted to do was an image with:
R: 255 everywhere
G: 255 everywhere
B: 255 everywhere
A: Mask

(Why? So that MipMap texture generation doesn't blend to black; don't worry about that. Should this be an issue that should be solvable on the game engine side? Sure, but I figured it'd be easier to quickly edit an image than diving into the Unreal Engine source code. But I guess I was wrong. It might also be easier to edit images via a hexeditor, bit by bit.)

I'm several hours deep into websearches and forum threads starting more than 4 years ago about how Affinity Photo can't help but modify the RGB channels when you modify the alpha channel, and the conclusion seems to be: "yeah, just follow this tutorial, it's perfectly possible". So I end up having to write my frustrations somewhere. Just in case someone at serif needs some further clues from customers that it might make sense to spend at least a tiny bit of attention to the use case of creating 2D assets for (real-time) renderers (e.g.: game engines). Fixing issues like this would make it a no-brainer to upgrade to v2.

[NotMyFault]

4 hours ago, RC-1290 said:

Describing this as perfectly possible feels a bit like presenting a command line interface and saying it's perfectly possible to create any texture you want.

It might be possible, but not without a PHD thesis on the subject.

Don't get me wrong: I appreciate that you put effort into documenting the behavior of this software. But this does not suddenly make it a feasible workflow.

All I wanted to do was an image with:
R: 255 everywhere
G: 255 everywhere
B: 255 everywhere
A: Mask

(Why? So that MipMap texture generation doesn't blend to black; don't worry about that. Should this be an issue that should be solvable on the game engine side? Sure, but I figured it'd be easier to quickly edit an image than diving into the Unreal Engine source code. But I guess I was wrong. It might also be easier to edit images via a hexeditor, bit by bit.)

I'm several hours deep into websearches and forum threads starting more than 4 years ago about how Affinity Photo can't help but modify the RGB channels when you modify the alpha channel, and the conclusion seems to be: "yeah, just follow this tutorial, it's perfectly possible". So I end up having to write my frustrations somewhere. Just in case someone at serif needs some further clues from customers that it might make sense to spend at least a tiny bit of attention to the use case of creating 2D assets for (real-time) renderers (e.g.: game engines). Fixing issues like this would make it a no-brainer to upgrade to v2.

I can’t see any issue to create such a document with Affinity Photo in under one minute. I probably missed some important piece of information. What is you actual question/issue?

add pixel layer

fill with white

add mask layer

[RC-1290]

17 hours ago, NotMyFault said:

I can’t see any issue to create such a document with Affinity Photo in under one minute. I probably missed some important piece of information. What is you actual question/issue?

add pixel layer

fill with white

add mask layer

I mean, that's similar to how I made my first texture. Except

1. It'll result in a pre-multiplied white texture (even as TGA)
2. How do you even edit a mask?

I usually figure it out again eventually, but it certainly isn't a matter of creating a mask, selecting it and painting into it, or pasting anything into it. The first challenge is selecting the mask, because it sure isn't the same as selecting the layer. Once you've managed, it won't show the mask itself, but the modified layer it masks. If you try to paste anything into it like normal, it'll create a new unrelated layer instead. For some reason the brush preview shows up in both the mask and the layer itself, which can get confusing. Sometimes only half such a preview shows up.

If you've managed to mask something, and export it, the result is pre-multiplied RGB channels. Apparently there is some filter that claims it'll let you do things that are not pre-multiplied, but that filter doesn't seem to affect the final result.

So I figured I'd follow this guide to hopefully find a way, but I don't even know where to start. Let me give you an impression of why this is hard by just describing going through it:

1. I'm not really trying to open a file, but maybe those steps are still important?
2. Why would I even need to create separate grayscale files?
3. There isn't a fill alpha button. There is a fill button on the background alpha channel, but it doesn't seem to do anything. I guess it's supposed to set the alpha to 1 on that layer
4. I had to look up what a channel mixer adjustment was, and the affinity documentation page I encountered for it, didn't even show where to activate it, or find it. I guess it in the Adjustment panel  in the same window as the layer panel, but... do I drag it somewhere? Does it actually activate when I click on that box, I don't see anything happening when I change any of those sliders. Oh, now that I switch back to the layers panel, there are suddenly a whole bunch of layers with Channel Mixer adjustments, what a weird UI. It helps to know we're talking about an adjustment layer.
5. "choose alpha channel". What? Drag the slider? Oh, no, that's in the next line. Oh wait, maybe they meant "In the channel mixer settings, under Output Channel, choose alpha in the second dropdown."
6. I interpret "Nest channel mixer layer in masking position to alpha layer" to mean the adjustment is indented to the side, under the alpha layer we created, but that seems... backwards? I'm not sure I'm doing this right. In what order does affinity apply things? Because in other examples, the masking shape was above the layer to mask, and mask layers seem to want to be underneath. Which way is it?
7. After following the steps and putting the alpha channel under the background layer, I figure it is time to start editing some things, only to find out the brush tool does nothing. The fill tool does nothing. I reset the channels stuff, just in case. Nothing. I unnest things. nothing. Selecting things? nah that wasn't it, and deselecting everything doesn't help either. It's not the color of the brush. And suddenly it works again, kind of.
8. Hmmm, this is really weird masking behavior, I guess "traditional alpha channel" has the definition here that everywhere the alpha channel is black, the "background" is visible, and everywhere it's white, it is hidden? That seems backwards, but okay, I guess we have different traditions.
9. Oh no, looks like Affinity's layer previews are being weird again, this time just not updating
10. Nevermind this, it'll be easier to just write some shaders.

Edited October 25, 2023 by RC-1290

[NotMyFault]

6 hours ago, RC-1290 said:

It'll result in a pre-multiplied white texture (even as TGA)

No, Affinity does not premultiply alpha in general. There might be some export formats which require premultiplied alpha, but PNG TIFF TGA doesn’t.

It depends on how you inspect the color values. You need to deactivate the alpha channel in channels panel to see the actual value. Values are multiplied for rendering, but not in the file.

 

[NotMyFault]

6 hours ago, RC-1290 said:

1. How do you even edit a mask?

 

Almost everybody in the forum including my person agrees that the process is complicated and we would be more than happy to get a more conventional UI / workflow. This has been discussed over and over again in multiple threads in the question section (or bug section).

You have replied to a tutorial, made by regular users trying to help others in knowledge of the current limitations.

Please stay on topic and avoid to clutter this thread with heated arguments or user/project specific topics. To get specific help for your project, simply open a new thread in the questions section and I will happily guide you through the process. 

Otherwise I may need to ask the moderators to close this thread.

Thank you for your understanding.

 

[RC-1290]

38 minutes ago, NotMyFault said:

Please stay on topic and avoid to clutter this thread with heated arguments or user/project specific topics.

I replied to the tutorial stating it's "perfectly possible". Then you asked for details. So I provided details about all the ways I get confused when trying to follow the tutorial. Unfortunately that also included aspects that you might find to be off topic. I figured they were relevant for context.

47 minutes ago, NotMyFault said:

Values are multiplied for rendering, but not in the file.

When I export a tga or a tiff of a white image from Affinity Photo, with some kind of mask, Unreal Engine 4.27, Blender 3.6.5 and Marmoset Toolbag 3 report black pixels in the masked out areas. How would you suggest this happens? All importers doing pre-multiplication?

Edited October 25, 2023 by RC-1290
Added version number to blender

[NotMyFault]

1 hour ago, RC-1290 said:

When I export a tga or a tiff of a white image from Affinity Photo, with some kind of mask, Unreal Engine 4.27, Blender 3.6.5 and Marmoset Toolbag 3 report black pixels in the masked out areas. How would you suggest this happens? All importers doing pre-multiplication?

Affinity deletes RGB values (sets to 0) when alpha is zero, but only when using the inherent alpha channel of a pixel layer.

this is not the same as premultiplied alpha. In case of alpha equals zero, the result is the same. It is not the same for all other alpha values.

this tutorial describes the exact process to avoid the loss of RGb values. By transferring the alpha channel into an seperate layer. 

[Robbin]

I followed it along but I must be missing a step as it always just comes out black, Im trying to set the alpha channel to 0 because a shader uses alpha to control something and I need it not to so it must be 0. However when I drag the greyscale layer to the top of the stack and rasterize to a mask it just makes the entire thing black.

 

 

 

 

 

 

 

 

 

 

 

 

 

Firstly editing the greyscale layer does nothing visibly so I dont even know if its working in the first place, the composite alpha channel stays white regardless of the greyscale value in the Alpha layer I made, It also just doesn't update the channel mixer when I set the values correctly, they just immediately reset to controlling the red channel when I reopen the window? 

When inspected with something like blender to see the rgb channels, they're all just white now? It also made the alpha channel itself white when it should be black?

This whole work around is god awful just because they don't care about a certain group of users trying to use a very often needed feature...

[NotMyFault]

I don't understand the layer stack. the channels mixer in your screenshot is nested to a rectangle - and above there is a group of layers. Normally the alpha channel is a pixel layer (when created by the tutorial). 

any chance you can upload the actual file?
and what editing do you want to achieve?

 

 

[Robbin]

I assume it should work regardless of what the channel mixer is nested on? since its all just the same data being modified no? All I want is to have the alpha channel be 0, whilst not messing with the RGB channels AlphaTest.afphoto

[NotMyFault]

If you want alpha to be zero, just add an empty mask layer on top.

the mask needs to be on top, otherwise it will get ignored.

 

[Robbin]

that still seems to not work, the image when inspected with blender looses all the data, its just all white... Is there some special export setting required?

 

[NotMyFault]

I checked again the tutorial on iPad with Photo V1 and V2. Unfortunately it is no longer working as before. Will test on desktop and update.

 

[NotMyFault]

on Mac the behaviour changed as below:

• The tutorial works as before, when having a mask which is not fully black, e.g. using a black to white gradient.
• If the mask is full black (all zero), the exported file gets wrong: either the zero mask is ignored (despite correct rendering of preview), and the exported file is 100% opaque, or the export is not recoverable (RGB channels zeroed out).

For your specific case requiring alpha to be zero for the complete canvas: this specific edge case is currently not possible with Affinity apps.

[Robbin]

wow truly amazing software... back to photoshop