Refine selection - unexpected "halo"

[norbre]

Hi,

I'd like to make a mask for a background replace of a portrait with Refine selection, but there is difference in the tool preview and the final result.

In the final result (masked layer,) there is some "halo" on the edge of the hair.
Here is the screenshot of the feature with overlay and black matte preview:
 

 

And here is a masked layer with black fill layer under it:

 

What did I wrong?

 

Thank you

[BarKeegan]

Hard to completely avoid that, especially when the subject has dark hair, and is placed on a dark bg. You might look into using Channels to make selections, it’s a process I use for selecting hair, fur etc

[James Ritson]

Hi @norbre, try setting your output on the Selection Refinement dialog to New Layer or New Layer with Mask.

The preview you are seeing includes colour decontamination—which means the background colour contribution to edge pixels is disregarded, eliminating the typical halo effect you would see. However, if you output to a selection or mask, Photo cannot perform this since it’s not manipulating the pixels of the layer—just masking them.

Using New Layer or New Layer with Mask allows the edge pixels to be decontaminated—hopefully this should solve your issue. In fact, if you choose New Layer with Mask and then turn off the mask layer, you will see exactly how the edge pixels and surrounding areas have been treated!