Different algorithm for masks+stacks?

[wyager]

I recently tried stacking some photos of the same subject with different lighting (to eliminate glare in post) and ran into an annoying issue: stacks don't play well with masks. If you attempt to mask out an item inside a stack using a soft brush, you get an extremely sharp boundary instead of the gradual transition you want. I can see why it was implemented this way, but I would like to suggest a different algorithm that's slower but at least useful (and should get the result people want and probably intuitively expect).

 

Let's say you have 5 layers in the stack. You set the stack mode to "median". However, in one area of the image, layer 3 is causing an object to look too dark. You'd like to mask out layer 3 in that area, and that area should look like layer 3 isn't being included (and gradually transition as you'd expect using a soft brush on your mask layer). You'd like to mask out layer 4 elsewhere because it's making something look too light.

 

To get the intuitive/desired behavior, you would calculate the stack 4 times (2^<number of layers with masks>):

 

A: Stack color channels of layers 1,2,3,4,5

B: Stack color channels of layers 1,2,4,5

C: Stack color channels of layers 1,2,3,5

D: Stack color channels of layers 1,2,5

 

Let a_i be the alpha channel of image <i>

 

a_A = a_3*a_4

a_B = (1 - a_3) * a_4

a_C = a_3 * (1 - a_4)

a_D = (1 - a_3) * (1 - a_4)

 

To get the final color output of the stack, you calculate A*a_A + B*a_B + C*a_C + D*a_D

 

Or something along those lines.