Jump to content
You must now use your email address to sign in [click for more info] ×

[AP] Refine selection "New layer with mask" broken


Imaginary

Recommended Posts

Yet another issue with selection refinement.
Refine selection with output to new layer is completely broken.
Steps to reproduce:

1. Use the selection brush to make a selection
image.png.52d061729459fddb0bd832f71c4a3a95.png

2. Refine selection witht the Matte brush
image.png.8987f1cca193556e8e2c45b6707c690b.png

3. Choose output to "New layer with mask" and apply
image.png.6e2c0fa15895281d6e7a901e6cf39434.png

4. Switch off the mask, the new pixel layer is completely destroyed. It shows the traces of the Matte brush
image.png.b585d0c39c580f0b44a2d305ecf54b6f.png


Sadly this is just one more issue with refining selections. Come on Serif, you should put a lot more effort in improving this function. PS is much more reliable here.
AP 1.7.3.481 Windows 10

Link to comment
Share on other sites

Opinion: The mask was an integral part of the output and process of refining the selection, so it doesn't seem fair to turn it off and then complain about the results. The new pixel layer is supposed to have that mask, to isolate your subject from everything else.

-- Walt
Designer, Photo, and Publisher V1 and V2 at latest retail and beta releases
PC:
    Desktop:  Windows 11 Pro, version 23H2, 64GB memory, AMD Ryzen 9 5900 12-Core @ 3.00 GHz, NVIDIA GeForce RTX 3090 

    Laptop:  Windows 11 Pro, version 23H2, 32GB memory, Intel Core i7-10750H @ 2.60GHz, Intel UHD Graphics Comet Lake GT2 and NVIDIA GeForce RTX 3070 Laptop GPU.
iPad:  iPad Pro M1, 12.9": iPadOS 17.4.1, Apple Pencil 2, Magic Keyboard 
Mac:  2023 M2 MacBook Air 15", 16GB memory, macOS Sonoma 14.4.1

Link to comment
Share on other sites

  • Staff

This isn't broken—the additional pixels you see have been treated for colour decontamination to optimise edge detail when the subject is cut out/isolated. As @walt.farrell mentioned above, you would not ordinarily see this because the mask is applied. If you were to take the destructive approach and only output as a New Layer you would never see these pixels, as they would simply be discarded.

Product Expert (Affinity Photo) & Product Expert Team Leader

@JamesR_Affinity for tutorial sneak peeks and more
Official Affinity Photo tutorials

Link to comment
Share on other sites

 
 
 
 
16 hours ago, James Ritson said:

This isn't broken

It may not be broken (or more accurately "as designed") but it is not intuitive or the expected behavior when working with the mask layer that is created during this operation 

E.g. If you need to further refine the mask by painting in white on the Mask layer you will reveal "artifacts" around the cut object that were never on the original image.

Which is why I prefer to duplicate the object then use the Mask output option rather than New layer with Mask

 

To save time I am currently using an automated AI to reply to some posts on this forum. If any of "my" posts are wrong or appear to be total b*ll*cks they are the ones generated by the AI. If correct they were probably mine. I apologise for any mistakes made by my AI - I'm sure it will improve with time.

Link to comment
Share on other sites

18 hours ago, walt.farrell said:

Opinion: The mask was an integral part of the output and process of refining the selection, so it doesn't seem fair to turn it off and then complain about the results. The new pixel layer is supposed to have that mask, to isolate your subject from everything else.

It is not a matterof fairness. I was not searching for issues, I just wanted to correct the mask after refining and stumbled across this. If anything, the software is not acting in a fair manner, because of a completely counterintuitive behaviour...
Besides I cannot see the point in having pixel layer and mask separated when both only work together...
If the mask only works with this exact pixellayer changes, it is just useless to have the separation.

17 hours ago, James Ritson said:

This isn't broken—the additional pixels you see have been treated for colour decontamination to optimise edge detail when the subject is cut out/isolated. As @walt.farrell mentioned above, you would not ordinarily see this because the mask is applied. If you were to take the destructive approach and only output as a New Layer you would never see these pixels, as they would simply be discarded.

Well, then why isn't colour decontamination an option in the refine dialogue? And why isn't it mentioned anywhere?

1 hour ago, carl123 said:

It may not be broken (or more accurately "as designed") but it is not intuitive or the expected behavior when working with the mask layer that is created during this operation 

E.g. If you need to further refine the mask by painting in white on the Mask layer you will reveal "artifacts" around the cut object that were never on the original image.

Which is why I prefer to duplicate the object then use the Mask output option rather than New layer with Mask

 

Indeed, I tried to improve the mask after refining..
Well, using output to mask doesn't give you the exact same result as previewed in the refine dialogue as I already pointed out about two years ago: [AP] Refine selection preview issue

Link to comment
Share on other sites

  • Staff
3 hours ago, Imaginary said:

Well, then why isn't colour decontamination an option in the refine dialogue? And why isn't it mentioned anywhere?

Indeed, I tried to improve the mask after refining..
Well, using output to mask doesn't give you the exact same result as previewed in the refine dialogue as I already pointed out about two years ago: [AP] Refine selection preview issue

I agree that it's not telegraphed very well, but essentially colour decontamination is linked to the issue @carl123 has mentioned as well as yourself in the above part I've quoted with using output to mask. Basically:

  • Output to Selection/Output to Mask is for when you intend to mask an adjustment or filter layer, or perhaps create mask bounds for brush work. Because the pixel layer is not modified, no colour decontamination can be applied, so you only get the refined selection.
  • Output to New layer/New layer with mask is for when you want to cut content out or isolate it from its background. Because the pixel content is modified, it means colour decontamination can be performed.

The confusion may arise because the preview modes (Overlay, Transparent, Black/White) apply the colour decontamination procedure, whereas if you output to an active selection or mask there is of course no way to apply this because it requires modifying the pixel content.

5 hours ago, carl123 said:

E.g. If you need to further refine the mask by painting in white on the Mask layer you will reveal "artifacts" around the cut object that were never on the original image.

If you need to do this, you are better off outputting to a selection or mask as you mentioned. The artefacts are a result of discarding the background colour contribution and using only the foreground colours over the matted areas—the intention is for these to always be hidden by the mask (or discarded if you just output to a new layer).

Should you choose to output as a new layer with mask, you also have other options to modify the mask:

  • Add a Curves or Levels adjustment, drag it over the thumbnail of the mask (this will place it beneath the mask). Set the channel target to Alpha. You can now control the matte blending.
  • Right click the mask layer and choose Refine Mask. Uncheck Matte edges (since the masking is already matted and you don't want to further matte the decontaminated areas) and set the preview mode to Transparent. You can then use Smooth, Feather and Ramp to further adjust the mask.

Product Expert (Affinity Photo) & Product Expert Team Leader

@JamesR_Affinity for tutorial sneak peeks and more
Official Affinity Photo tutorials

Link to comment
Share on other sites

Thank you, James, for those detailed explanations of the technical background. It starts making sense now.
Nevertheless I think that colour decontamination should be an option and have an intensity slider like the other parameters.

And I also know that documentation is just an imposition for software developers, but the the "Refine selection" part really lacks a lot of information about what is going on when particular options are chosen. So it really needs a thorough revision. And if only to avoid false impressions...

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...

Important Information

Terms of Use | Privacy Policy | Guidelines | We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.