Anti-aliasing artifacts after refining selection to mask

[BLVCKFEINT]

apologies if this has been posted elsewhere! i haven't been able to find anything in the v2 forums via searching.

i'm having issues getting an exact selection refinement output to a mask. the refinement window shows exactly what i want, with clean edges and no artifacts anywhere. however, when i have it create a mask out of the refinement, i can still see little green artifacts along the edge of the model.

am i missing a step here? my thought would be that whatever the refinement window shows me is what i'd get, unless there's some other factors i have to adjust. there's nothing super complex here that i'd need the brush for either, it's just a bunch of polygons.

does the method of initial selection matter? i've been selecting the green screen background, then inverting it so i have the model selected, so i can get it isolated on a transparent background. i have no issue adjusting the mask itself to get it perfect, i'm more just wondering if this is user error or an issue with the software.🤔

i'm using photo 2.0.3.

[lepr]

16 minutes ago, BLVCKFEINT said:

i'm having issues getting an exact selection refinement output to a mask. the refinement window shows exactly what i want, with clean edges and no artifacts anywhere. however, when i have it create a mask out of the refinement, i can still see little green artifacts along the edge of the model.

The preview shows what you'll get if you choose New Layer With Mask for the output, instead of just Mask.

[RichardMH]

Does unchecking Matte edges help?

[NotMyFault]

One easy way to remove color spill on edges is to add an HSL adjustment: desaturate and adjust lightness. Then invert mask, and paint in where needed with white brush. This gives far better control and is faster than fiddling with selections and refine tool.

The “color decontamination” of using “new layer with mask” as output can lead to severe brightness shift as another user found.

[BLVCKFEINT]

7 hours ago, ,,, said:

The preview shows what you'll get if you choose New Layer With Mask for the output, instead of just Mask.

this worked like a charm, thank you!

7 hours ago, RichardMH said:

Does unchecking Matte edges help?

it did, actually! i just noticed it was actually adding some unwanted color to a few of the edges of the render as well.

6 hours ago, NotMyFault said:

One easy way to remove color spill on edges is to add an HSL adjustment: desaturate and adjust lightness. Then invert mask, and paint in where needed with white brush. This gives far better control and is faster than fiddling with selections and refine tool.

The “color decontamination” of using “new layer with mask” as output can lead to severe brightness shift as another user found.

i'll advance further into this project tomorrow, so if i have any issues, i'll keep this in mind! thank you!

[laurent32]

On 2/2/2023 at 11:18 PM, NotMyFault said:

One easy way to remove color spill on edges is to add an HSL adjustment: desaturate and adjust lightness. Then invert mask, and paint in where needed with white brush. This gives far better control and is faster than fiddling with selections and refine tool.

Hi @NotMyFault, could you send a short video of the steps, I'd like to better understand that easy way ?

Thanks.

[kboone]

I've found that Output with 'New layer with mask' does not behave the same as outputting with 'mask'.

Afterwards, whenever I try and paint in areas of the mask from "new layer with mask" , I get weird artifacts and the mask is not filled in. 

However, with just the output from 'mask', I can correct areas that I've missed.

Anyone else experience this?

[NotMyFault]

55 minutes ago, kboone said:

I've found that Output with 'New layer with mask' does not behave the same as outputting with 'mask'.

Afterwards, whenever I try and paint in areas of the mask from "new layer with mask" , I get weird artifacts and the mask is not filled in. 

However, with just the output from 'mask', I can correct areas that I've missed.

Anyone else experience this?

Simply deactivate the mask, and you will see the difference. New layer with mask will do some „color decontamination“ on the new layer, which is destructive. So if you want to paint in additional areas, never use „new layer with mask“.