Affinity Photo - Masking artifacts - what caused them and how to remove them?

[Ken Hjulstrom]

Hi,

 

I've encountered an odd problem in Affinity Photo, in which artifacts, which appear to be due to some sort of masking error, are appearing on an exported TIFF file.  These artifacts don't seem to be visible in the Photo app, so I'm not sure how to fix them.  The example Photo file is one in which I manually masked out the original background, which was a shade of white.  When I do background removal, I create a fill layer that has a sharply-contrasted color to the subject, I select the mask layer, and I then draw on the mask layer, using the contrasted fill color to let me know what I've masked off and what I haven't.

I've attached the following files:

• #155_CoverImage_Transparent_FixedMaskVisibility_24Sep24.tiff:  The output file from Photo
• #155_CoverImage_Transparent_FixedMaskVisibility_24Sep24.afphoto:  The Photo file
• Screenshot-AffinityPhotoExportSettings:  Screenshot of Photo export settings to TIFF
• Screenshot-ExportedImageDisplayedInPreviewApp:  Screenshot of TIFF file displayed in the MacOS Preview app, showing the artifacts (most clearly in the area within the sewing machine arm)
• SmudgesOnPressReadyVersion_25Sep24:  Publisher file containing the TIFF file against a colored backgrouns
• Screenshot-AffinityPublisherExportSettings:  Screenshot of Publisher export settings to PDF

Clearly, this is a Photo issue and not a Publisher issue, though in Publisher, the artifacts aren't visible in the default "Digital-Small Size" PDF export settings, though they are in the default "Press Ready" export settings.

Assuming this isn't a bug, could someone explain how this would have happened in the first place, and also, how can I fix this in a non "hit-or-miss" manner, since I haven't found a way to view these artifacts in Photo.

Thank you,

Ken

#155_CoverImage_Transparent_FixedMaskVisibility_24Sep24.afphoto #155_CoverImage_Transparent_FixedMaskVisibility_24Sep24.tiff SmudgesOnPressReadyVersion_25Sep24.afpub

[steday]

Hello

If I understand clearly what you ask :

It is your masking layer that is not perfect. In the attached screen (See the mask), if you display the mask (ALT + click on Mac) you can see that the mask is not totally black in the middle, so it doesn't totally mask the corresponding part of the image.

To resolve this, you can use a brush with black color to paint the bad zones of the mask (see attached Paint the mask). Paint black on a mask hide the corresponding zone of the image, painting white reveal the corresponding zone of the image.

See attached : See result on grey background (for better contrast).

Hope it could help

 

[NotMyFault]

To make such areas better visible, you can do this:

1. add a curves adjustment
2. choose alpha channel
3. choose right node at 1/1 position
4. drag it left to 0.1 / 1.0 position

now all imperfections (almost black but not fully black) become much brighter and easy to spot.

If you need hard zero or one mask, you can use a curve adjustment to heal the issue.

1. reset the curves adjustment
2. add a new node at 0.5 / 0.5 (middle)
3. drag the node fully down right to 1.0 / 0.0

you can now see that the white areas are faulty, too.

Honestly, I recommend to start over the masking process and do it again so you can develop the perfect workflow:

• blacks are fully black
• whites are fully white
• transitions at hard edges have the correct gradient, either steep or shallow
• transitions at textured edges like hair, carpet etc are textured.

this requires to carefully select suitable brush (basic round brush or brushes from masking category) and brush settings (color, opacity, hardness).

In general I recommend to not use the refine mask functionality when you have hard edges as it will create spills and spoil all hard edges.

 

[NotMyFault]

Below you can find the alternative approach using a vector mask made by pen/node tool. I added a 0.5px Gaussian blur to get softer edges.

Took me about 15 minutes (see timestamp of my posts)

next step would be to adjust brightness at edges based on lightning (remove bright glare coming from one direction).

 

#155_CoverImage_Transparent_FixedMaskVisibility_24Sep24-NMF.afphoto

[Ken Hjulstrom]

41 minutes ago, steday said:

Hello

If I understand clearly what you ask :

It is your masking layer that is not perfect. In the attached screen (See the mask), if you display the mask (ALT + click on Mac) you can see that the mask is not totally black in the middle, so it doesn't totally mask the corresponding part of the image.

To resolve this, you can use a brush with black color to paint the bad zones of the mask (see attached Paint the mask). Paint black on a mask hide the corresponding zone of the image, painting white reveal the corresponding zone of the image.

See attached : See result on grey background (for better contrast).

Hope it could help

 

Hi steday,

Yes, my problem is that my mask isn't perfect, but the imperfections are very minor, often seeming to be individual pixels.  I'm not sure exactly what causes these, but over the course of creating a large mask, as I did here to remove the background from the image I posted, it's possible that I may have inadvertently drawn or pasted a color other than pure black or pure white onto the mask.  The problem that resulted in my post here is that for this particular image, the mask errors were so small that they were invisible to the eye in Photo and Publisher, with the result that I didn't realize there was a problem until I received notification from the print shop that things didn't look right.  It turned out that the masking errors didn't reveal themselves until I generated a high-def PDF, which did clearly reveal what was going on.

I finally figured out that I could use the Flood Select Tool to show me where the inconsistenties in the mask layer were.  With the mask layer selected, the Flood Select Tool would reveal the mask errors as tiny selected (non-black) regions within the much-larger black areas.  By going through a rather tedious sequence of using the Flood Select Tool to reveal the error locations, switching to the Paint Brush Tool, positioning it over one of the selected areas, turning off the selection, and then painting over the error location, I eventually cleaned up the mask.  It looks like NotMyFault's post here is a more straightforward approach to patching up mask errors.  I'm wondering if my use of the Paint Brush Tool with less than 100% hardness might be introducing these errors.  I typically use less than 100% hardness when following object outlines, as I get a better transition between the subject and the (now transparent) background.

Thanks for the information,

Ken

[Ken Hjulstrom]

37 minutes ago, NotMyFault said:

To make such areas better visible, you can do this:

1. add a curves adjustment
2. choose alpha channel
3. choose right node at 1/1 position
4. drag it left to 0.1 / 1.0 position

now all imperfections (almost black but not fully black) become much brighter and easy to spot.

If you need hard zero or one mask, you can use a curve adjustment to heal the issue.

1. reset the curves adjustment
2. add a new node at 0.5 / 0.5 (middle)
3. drag the node fully down right to 1.0 / 0.0

you can now see that the white areas are faulty, too.

Honestly, I recommend to start over the masking process and do it again so you can develop the perfect workflow:

• blacks are fully black
• whites are fully white
• transitions at hard edges have the correct gradient, either steep or shallow
• transitions at textured edges like hair, carpet etc are textured.

this requires to carefully select suitable brush (basic round brush or brushes from masking category) and brush settings (color, opacity, hardness).

In general I recommend to not use the refine mask functionality when you have hard edges as it will create spills and spoil all hard edges.

 

Hi NotMyFault,

Thanks for posting this approach; I'll probably try this the next time I need to do some background removal masking.  This is work that I'm doing for a magazine, in which the images are submitted by article authors, and I quite often am presented with images that aren't ideal (odd lighting, awkward attempts at background removal, etc.), which makes the magazine layout process interesting and sometimes challenging.

Ken