White edge on overlays and selections

[Delta5]

Until recently I've used mainly Nikon Capture NX2 and DxO for RAW processing - both have 'colour control point' tech to select areas for adjustments.

I'm now learning AP (v1.6.5.135), and right now trying to get the hang of the overlay and selection methods - but I'm getting the white edges shown in the attached screenshots. The original is a RAW (.NEF) image selected purely as a sample for testing.

In the first instance I used Develop Persona and the Overlay Paint tool to select the sky with edge aware. When I painted in the overlay exposure adjustment I got a white edge on the overlay and was unable to find a way to remove it. (Ignore the blotchy uneveness of the paint tool - that's another matter which I'll address separately!)

In the second I used Photo Persona and followed one of the AP video tutorials to create an alpha channel selection (with refine) which I then applied to a Brightness and Contrast adjustment layer (for the land) and inverted to a Curves adjustment layer (for the sky). The nearest I've come to fixing this one is by editing the (land) layer blend options - this seems to work up to a point but still doesn't quite eliminate all of the edge.

Is there a way to avoid creating the white edge in the first place (particularly for the Overlay Paint tool which I'd use a lot), or a better way to remove it than layers and blend options?

TIA

[firstdefence]

In Photo Persona you probably need to Refine the edge to get a tighter fit to the cliff.

To replace skies there is a neat trick with layer blend ranges, see this video:

 

[Dan C]

Hi Delta5,

Welcome to the forums

Could you possibly provide the sample image you're using so I can look into this for you?

[Delta5]

Hi FirstDefence - thanks for the Blend Ranges vid. I had a quick go at that, with some success, and I'll take a deeper dive asap and see where it goes. I'm really more bothered about the issue with the Overlay as I'm likely to use that much more.

Dan C - thanks for the welcome and offering to help - I'll have to upload the full image from work in the morning (my connection here is too flakey).

D5

 

[Delta5]

Hi Dan C - here's the original .NEF

Thanks

2018_09_13_6964_AP_trial_copy.NEF

[Dan C]

Thanks for that

I've created the following using the overlay paint tool in the Develop Persona, set to 100% hardness with edge detection on and then dropping the exposure level of the sky area painted. 

This still created some white haloing, and I much prefer the workflow in the Photo Persona for this style of editing.

I used the smart selection brush to select the sky area, then duplicated this pixel area. I then inverted the selection and repeated the duplication, so I have 3 pixel layers total, the main image, a sky layer and a land layer.

With the main background image at the bottom of the stack, I set both the ground layer and sky layer to Multiply blend mode, changing the opacity of the ground layer to 50%, producing the following result:

I've attached a copy of the .afphoto file with history so you can pan back through my document to see this in action. (Ignore the adjustment layers created and deleted, I was playing around to see what produced the best result )

2018_09_13_6964_AP_trial_copy photo.afphoto

[MEB]

Hi Delta5,
Welcome to Affinity Forums
To prevent those halos from appearing use a larger brush (200px or bigger) and pass a few times around the edges/boundaries without letting the cross in the centre of the brush approach too much to the edge - make sure Edge Aware is ticked. Since we don't have control over the flow of the brush passing it a single time is not enough to fully select around the edges, leaving those "glows" around.

While passing a few times with the brush it's possible you may overdo a little sometimes leaving the edge with a thick outline. See image below (exaggerated a little):

To correct these cases, switch to the Erase Overlay Tool set the hardness to 0% (you may also want to reduce the brush size a little) and from the other side of the boundary (the one you are NOT selecting) paint again near the edges without letting the cross in the centre touch the boundaries to smooth out the thick edges a little - the goal here is ensure that they are neither too thick/dark nor entirely deselected/affected (leaving a white outline around the edge) - the more neutral they look (no darker nor lighter outline around the edge) the better. See below. Here is not enough:

Below is fine/correct:

The end result hopefully without halos/glow should look like this:

Please ignore the adjustments/colors as they were intended to make the sky standout again the land (the sky is way to blueish and the water wasn't adjusted to reflect that so all looks a little too fake specially where the water meets the sky) but the point was to show how the overlay can be created without the halos/glow. I'm not attaching the file here because there's no way to save it while in Develop Persona and switching to Photo Persona bakes up all adjustments/overlays so there's no way to check them.

[Ballyshannon]

Nice job MEB!

[Delta5]

Dan C, MEB

Thanks very much for taking time to help me with this - I now need to find some slack time when I'm not dead beat to play with these methods some more - hopefully this w/e.

Then I also need to figure out why the Develop Overlay paint tool lays down a blotchy uneven pattern instead of a smooth selection (the first image I uploaded shows the effect when an adjustment is applied to the selection!). Another example screenshot below FYI. This was at about 50% hardness. The solid selection on the left is where I went over it a few times. The effect is worst with edge aware checked, but also visible without edge aware at 100% hardness. Maybe this is just the way AP works but if not and you have any ideas on this please let me know.

Cheers

D5

 

 

[MEB]

Hi Delta5,
That "blotchy uneven pattern" is due to a low brush flow as I mentioned above. Since we don't have Flow control (like the Paint Brush Tool in Photo Persona for example) for the overlays in the context toolbar (in Develop Persona) you have to pass the brush multiple times over the same area until it becomes solid. I will fill an improvement request for this.

[Delta5]

Thanks MEB, good to know.