Machine Learning: Select Subject

[Patrick Connor]

Apps: Photo
Platforms: macOS, Windows and iPad

A Machine Learning Feature

As you will see there are two new features which use machine learning models for automatic object and subject selection. These features are optional and require downloading of the relevant models for them to work (instructions included in this post). We want to make sure it is clear that these are installed as pre-trained models and they do not use any of your own data for further training. Furthermore these operations all work ‘on device’ meaning none of your data leaves your device at any time.

Select Subject

Where to find it

"Select" Menu > "Select Subject"

Initial Setup

Automatic Select Subject relies on Machine Learning Models, which need to be installed prior to using the feature.

 

To install the Machine Learning Models, you will need to visit Settings > Machine Learning Models. Both Segmentation and Saliency need to be installed. Since all machine learning inference happens on the device itself, the models need to be downloaded and stored locally. These models can be quite large in size, making it impractical to include them in the initial app installation. Users also have the choice to uninstall these models to free up space if they are not needed.

Segmentation

The Segmentation model allows Photo’s Object Selection Tool to create precise, detailed pixel selections from pixel layers or placed images.

Saliency

Saliency, installed alongside Segmentation, allows Photo to understand what is visually significant on a pixel layer or placed image. It is required for the new one-click Select Subject to work.

How to use it

Available from top menu’s Select > Select Subject, the feature analyses the scene and selects what it considers to be the main subject. This operation can be recorded as a Macro to aid setting up foreground/background retouching workflows for example.

Select Subject is available both on Affinity Photo 2 desktop and iPad.

COMPATIBILITY: Sorry but due to the functions we need to run the Object Selection and Subject Selection functionality, the Machine Learning features only available on Apple Silicon iPads and macOS devices (and those need to be running on a recent macOS, as Catalina, Big Sur and Monterey do not support the required calls). Machine Learning will run on both Windows x64 and Arm64 hardware running Windows 10 or Windows 11.

[Affinity Info Bot]

Known Issues in this feature
A list of unresolved issues for this feature, reported by users

• AF-3596 - iPad > App Settings > Machine Learning Models > No progress bar shows when installing models
• AF-4763 - Machine Learning section of iPad Settings is truncated when in Portrait
• AF-4815 - The Quick Mask Bounding Box is Incorrectly Positioned for Selections on Non Facing Page Spreads
• AF-4856 - Publisher and Photo Crash When Using Select Subject on a Placed PDF Document
• AF-5095 - Memory in use increases when using Object Selection or Selection Subject and is not released until session is closed

Released Fixes
A list of issues for this feature, available in the current beta build

• AF-4512 - Object Selection Tool: Selections are off in Photo Persona
• AF-4812 - Select Subject Fails to Select Subjects on Right-hand Facing Pages Following the First Page in a Publisher Document...
• AF-4720 - Crash after installing Machine Learning Models when Install is clicked multiple times
• AF-4746 - App crashes when using Select Subject on a blank document
• AF-4845 - App crashes after downloading Machine Learning models on Catalina, Big Sur and Monterey

[n_shcherbakov]

Wow! 

[joe_l]

In the first post Machine Learning is said to be meant only for AP, but APu has the same settings.

[Patrick Connor]

5 minutes ago, joe_l said:

but APu has the same settings.

Only if you are an owner of Photo (and Publisher) We do not normally mention in Publisher features of Photo and Designer that only appear because of StudioLink

[anto]

Publisher crashes when select Photo Persona -- Select -- Select Subject

 

 

[Hangman]

Using Select Subject after deselecting a selection created using the Object Selection Tool causes Affinity Photo to crash...

Steps to Reproduce

1. Open a file in Affinity Photo
2. Use the Object Selection Tool to make a selection
3. Deselect the selection Cmd D (Ctrl D)
4. Select the Move Tool
5. Click off the image so it’s no longer selected
6. From the Select menu choose Select Subject
7. Photo crashes

Note: This affects both Mac and Windows and the Photo Persona in Publisher

[smadell]

I originally posted this in a different “bug” section, but am copying it here where it is more appropriately placed.

Just installed the iPad Photo 2.6 beta and installed the necessary ML files. Since this was exactly the upgrade I was hoping for, I immediately tested it. Both the “Select Subject” option and the Object Selection tool, when chosen, will result in a few seconds of apparent “thought” followed by an unceremonious crash. The app goes black and closes back to the iPad home screen. I tried to provide a screen recording, but the crash seems to have crashed the screen record process as well.

I have re-booted the iPad, force quit and re-started the app, and neither of these affected the outcome.

I’ve not yet tried the new machine learning selection options on my desktop. I’m still hoping!

[Daniel Gibert]

Just tested it on some actual work photos and whoa! this could save us tons of work. And I love the approach by local computing and local trained model. That is the way to do it.

[dcarvalho84]

6 hours ago, Hangman said:

Using Select Subject after deselecting a selection created using the Object Selection Tool causes Affinity Photo to crash...

Steps to Reproduce

1. Open a file in Affinity Photo
2. Use the Object Selection Tool to make a selection
3. Deselect the selection Cmd D (Ctrl D)
4. Select the Move Tool
5. Click off the image so it’s no longer selected
6. From the Select menu choose Select Subject
7. Photo crashes

Note: This affects both Mac and Windows and the Photo Persona in Publisher

 

I can confirm this, but i think it's just because we have the layer deselected. I we repeat all those steps but before select subject you click the layer in layers panel it won't crash, at least not for me.

[Mithferion]

On 10/21/2024 at 6:15 AM, Patrick Connor said:

We want to make sure it is clear that these are installed as pre-trained models and they do not use any of your own data for further training.

Hi!

So, considering this, I have some questions and I'd like to know if there is something regarding this that you can share:

• Are going to keep on training these models?
• If so, are going to make them available independently of the Affinity Photo's version?
• Maybe some users would like to share their pictures for further training, would you allow that kind of involvement?

Best regards!  

[Patrick Connor]

41 minutes ago, Mithferion said:

• Are going to keep on training these models?

We are using pre-trained libraries. The library authors may go on to improve their model with more training yes, and we would choose whether to make that available to you if it gives significantly better results.

41 minutes ago, Mithferion said:

• If so, are going to make them available independently of the Affinity Photo's version?

We can do that but it's more likely to be done optionally in an update.

41 minutes ago, Mithferion said:

• Maybe some users would like to share their pictures for further training, would you allow that kind of involvement?

That's not relevant, seeing as we are not doing the training (see my first answer).

[Andy05]

Great first step! But I hope there will be quite some training. I tried various images and tested against other apps' features and Photo's select objects seems to be a bit behind all of them. Even in illustrations (testing just for testing purpose) with extreme contrast, the result is a bit disappointing:

 

[EmT]

17 hours ago, smadell said:

Both the “Select Subject” option and the Object Selection tool, when chosen, will result in a few seconds of apparent “thought” followed by an unceremonious crash.

Hi @smadell this is a known issue, if you double tap install or install them both at the same time then this can cause the app to crash, if you install one then the other it should be fine. 

[kaffeeundsalz]

41 minutes ago, Andy05 said:

Even in illustrations (testing just for testing purpose) with extreme contrast, the result is a bit disappointing

You might think that illustrations should be easy, but from my testing, results are almost always better with real photos. Pretty sure this has to do with what the models were trained for.

[dcarvalho84]

1 hour ago, Andy05 said:

Great first step! But I hope there will be quite some training. I tried various images and tested against other apps' features and Photo's select objects seems to be a bit behind all of them. Even in illustrations (testing just for testing purpose) with extreme contrast, the result is a bit disappointing:

 

From the quick test i've done so far, i honestly think that for this kind of images the object selection tool provides better results because you can see the segments and choose the ones you want.

[Juhani]

Played around with the new feature a bit and compared it to a couple of other tools that I regularly use, Pixelmator Pro and Photoshop, so thought I'd share the result with you as well. All of them are done using one click Select Subject method without following up with any refinements. Overall Affinity seems the most crudest of them at this point, with some exceptions, but that's understandable given this is the very first version of this function.

Dark hair on dark busy background
Photo by Arun Prakash on Unsplash

Original, Pixelmator Pro, Photoshop,  Affinity

 

Dark hair on light even background
Photo by Onkarphoto on Unsplash

Original, Pixelmator Pro, Photoshop,  Affinity

 

Blond hair on blue sky
Photo by Ayo Ogunseinde on Unsplash

Original, Pixelmator Pro, Photoshop,  Affinity

 

Clean edge subject + object on light background
Photo by Eric Nopanen on Unsplash

Original, Pixelmator Pro, Photoshop,  Affinity

 

 

 

Tree on blue sky background
Photo by niko photos on Unsplash

Original, Pixelmator Pro, Photoshop,  Affinity

[kaffeeundsalz]

53 minutes ago, dcarvalho84 said:

From the quick test i've done so far, i honestly think that for this kind of images the object selection tool provides better results because you can see the segments and choose the ones you want.

Well, not necessarily. Here's an example of what the Object Selection Tool thinks are the correct boundaries of an illustration part:

It's not always that bad, but this is where my assumption comes from that the models aren't really tailored to illustrations. Which makes sense because the problem above shows that it's just a bad idea to try to solve everything with AI just because it's available now: I'd certainly be much, much faster doing the selection with the Selection Brush Tool  - or with the Flood Selection Tool on the plain-colored background. This might apply to many illustrations and especially for the example @Andy05 provided above. If it weren't just for testing purposes, you'd use other tools to get the job done, and these have been available in Affinity Photo for a long time.

[kaffeeundsalz]

@Juhani Wow, that's a really insightful comparison. To be honest, I would have expected at least Photoshop to deliver better results, but no way. Especially the tree is a mere catastrophe with all three applications. This is why I took your experiment just one step further:

Left: Select Subject, then use the Flood Selection Tool in Subtract mode to get rid of the remaining background artifacts.

Right: Isolate blue channel, Select > Tonal Range > Select Highlights, Select > Invert Pixel Selection, Select > Refine Edges.

As you can see, the first approach starts with the new Select Subject feature and leads to a much better result with just one additional editing step. The second approach avoids Select Subject entirely, again showing that you don't necessarily need AI (but the right tool for the right job instead).

This is a good demonstration that Select Subject can be a great starting point for masking even if the initial results appear mediocre at best. The left mask took less time to create, at least when I subtract the processing time for the Select Subject feature. Given that my machine is pretty old, I'd expect the advantage to be even more obvious had I done this on a more modern computer.

[Juhani]

47 minutes ago, kaffeeundsalz said:

As you can see, the first approach starts with the new Select Subject feature and leads to a much better result with just one additional editing step. The second approach avoids Select Subject entirely, again showing that you don't necessarily need AI (but the right tool for the right job instead).

Yeah, for sure, in the real world I'd probably combine Live Hue Range with Luminosity Range masks together with the tree example to keep everything non-destructive, which btw are just fantastic selection tools and neither Photoshop nor Pixelmator Pro has anything like it, but the point of the experiment was to specifically compare these applications in terms of their Select subject capabilities and show that Affinity has still a lot of work to do with its Select Subject models.

[fde101]

2 hours ago, Juhani said:

Original, Pixelmator Pro, Photoshop,  Affinity

If I am looking at it correctly, it seems to me, at least "subject"ively, that PixelMator Pro currently gives the best results of the three, except with the tree.  Not perfect, though.

Still, with Photo in beta and this being a first pass, it is not a terrible showing.  I do hope they improve the models going forward.

I do think that the support for spreads of more than 2 pages in Publisher is the real headline feature of this release, though.

[carl123]

1 hour ago, Juhani said:

But the point of the experiment was to specifically compare these applications in terms of their Select subject capabilities and show that Affinity has still a lot of work to do with its Select Subject models.

I'm not sure Affinity has much control over the way the ML models work as that may be in the hands of those who created those models

I think for complex images the ML models will never be enough and you will always need to refine those selections

For that reason, I think Affinity should concentrate on improving its own refine operation which can be good with some images but extremely problematic on others

Below are a couple of examples where it (refine) works reasonably well and fast

PS I'm not sure if Photoshop or Pixelmator has refine capabilities but it would be nice to see a comparison of them compared to Affinity one day

 

 

 

[Andy05]

2 hours ago, kaffeeundsalz said:

I'd certainly be much, much faster doing the selection with the Selection Brush Tool  - or with the Flood Selection Tool on the plain-colored background. This might apply to many illustrations and especially for the example @Andy05 provided above.

3 hours ago, kaffeeundsalz said:

You might think that illustrations should be easy, but from my testing, results are almost always better with real photos. Pretty sure this has to do with what the models were trained for.

Yes, I came to pretty similar results like @Juhani in his comparison for photos (I also had Luminar Neo for testing on this). The illustration was just for further testing purposes in order to see if the algorithm is mainly focusing on stark contrast in images. 

Of course, other tools are way superior for masking in such cases.

Therefore, I totally agree with @carl123 - in conjunction with some additional tweaking (mask refining), the new feature is quite useful. And probably improving the mask refinement feature is the way to go in the future.

BTW, I am actually one of them "AI can't (and shouldn't) do everything for you, but it can be a great aid during your workflow"-persons.

[CM0]

1 hour ago, carl123 said:

For that reason, I think Affinity should concentrate on improving its own refine operation which can be good with some images but extremely problematic on others

Yes, this would probably yield the best result. As seen from the competitor products, ML isn't pure magic. Nobody has a solution that works perfectly. Having a combination of effective tools is still required.

[kaffeeundsalz]

51 minutes ago, Andy05 said:

BTW, I am actually one of them "AI can't (and shouldn't) do everything for you, but it can be a great aid during your workflow"-persons.

Yeah, me too! Hope I've made this clear with my above post.