Anti aliased edges when filling a selection?

[Texel]

I'd like to fill a selection boundary and have the alpha channel set based on pixel coverage.

I have tried the 'feather' option. That is not what I need.

Steps:
1. Create a small pixel layer on empty RGBA8 image (eg. 64 or 128 pixels square)
2. Create selection rectangle
3. Rotate selection rectangle 45 deg eg. by Toggle Quick Mask and rotate handle.
4. Try fill selection via Fill & Paint Brush tools.

I expect the result to look the same as if using the Rectangle shape tool and rotating it.

Attached image shows rectangle shape tool result bottom left and filled selection result top right.

[DWright]

Can you please confirm if you have the Force Pixel alignment option enabled in the context toolbar as this will reduce the amount of anti aliased on the rotated object.

[Texel]

Force Pixel alignment was on. I tried both on and off, which made no difference. I also tried both on and off before creating and rotating the selection for the test steps.

[Old Bruce]

On 5/16/2022 at 11:38 PM, Texel said:

I expect the result to look the same as if using the Rectangle shape tool and rotating it.

If you are expecting a pixel grid to not show then it is impossible to achieve when working with pixels. I think you may be using Photo correct? So you are seeing the pixelization of the rotated rectangle and you want sharp edges on the pixel layer too, am I correct?

On 5/16/2022 at 11:38 PM, Texel said:

3. Rotate selection rectangle 45 deg eg. by Toggle Quick Mask and rotate handle.

Why use Quick mask? I just choose the Move tool and there are handles applied to the selection and I can move and rotate using the move tool or the Transform panel.

I get nice sharp edges if I am doing the work on a new empty pixel layer.

[iconoclast]

To assess the grade of pixelisation of a raster object, you have to watch it with a zoom factor of 100%. If you choose a higher zoom factor, it will of course look more pixelated than it really is. A raster object will never look as sharp and smooth edged as a vector object. That's one of the main reasons why vector graphics exist.

[JimmyJack]

On 5/17/2022 at 2:38 AM, Texel said:

I'd like to fill a selection boundary and have the alpha channel set based on pixel coverage.

I have tried the 'feather' option. That is not what I need.

Steps:
1. Create a small pixel layer on empty RGBA8 image (eg. 64 or 128 pixels square)
2. Create selection rectangle
3. Rotate selection rectangle 45 deg eg. by Toggle Quick Mask and rotate handle.
4. Try fill selection via Fill & Paint Brush tools.

I expect the result to look the same as if using the Rectangle shape tool and rotating it.

Attached image shows rectangle shape tool result bottom left and filled selection result top right.

 

18 hours ago, Texel said:

Force Pixel alignment was on. I tried both on and off, which made no difference. I also tried both on and off before creating and rotating the selection for the test steps.

I can confirm this behavior. FPA on and off.

It's like the rectangle marquee ignores the antialias checkbox.
(while we're on it... why else would the rectangle marquee need an antialias checkbox if not this???)

[Texel]

To clarify, this is not about the rectangle shape, or moving and selecting rectangles.  That was just my simple repro steps.

I'm talking about filling or painting inside a selection boundary. The is no coverage anti aliasing. A pixel is either colored or not.

[JimmyJack]

45 minutes ago, Texel said:

To clarify, this is not about the rectangle shape, or moving and selecting rectangles.  That was just my simple repro steps.

I'm talking about filling or painting inside a selection boundary. The is no coverage anti aliasing. A pixel is either colored or not.

You mean any selection shape period? Or on any selection shape that's been rotated (via quick mask or not)?

Because if it's any shape, then I have to ask (yup, here it comes....sorry) do you have the anti-aliasing box in the context toolbar checked when making your selection?
Because any other shape: oval or freeform does have the anti-aliasing for me when I use paint bucket fill, brush/paint fill, drop down "fill with *whatever* color".

Now, once a selection shape has been rotated, then the aliasing at times becomes very very ugly (a problem.... maybe a whole different thread to start).... but it is still there. The rectangle was the only selection that didn't.

[Texel]

@JimmyJack Interesting, I hadn't isolated the issue.

It does not occur with the circle selection, even though the circle selection shows a pixelated border rather than smooth as I'd expect. The anti alias checkbox works with the circle selection tool exactly as expected an desired.

The issue does appear to relate to rotation of the selection.

Steps:

1. Draw a shape. (Eg. white on black, solid edges)
2. Select shape (Eg. Flood select)
3. Resize and rotate selection, not shape. (I'm using Quick Mask as I can't see another way)
4. Flood fill or use Brush to fill in selection.

[JimmyJack]

48 minutes ago, Texel said:

@JimmyJack Interesting, I hadn't isolated the issue.

It does not occur with the circle selection, even though the circle selection shows a pixelated border rather than smooth as I'd expect. The anti alias checkbox works with the circle selection tool exactly as expected an desired.

The issue does appear to relate to rotation of the selection.

Steps:

1. Draw a shape. (Eg. white on black, solid edges)
2. Select shape (Eg. Flood select)
3. Resize and rotate selection, not shape. (I'm using Quick Mask as I can't see another way)
4. Flood fill or use Brush to fill in selection.

Yeah, I'm agreeing with you. AND confirming the behavior.

The aliasing setting and straight edges (horiz/vert) aren't working together when rotated.

(And I still think the other rotated aliasing...... well..... stinks)

[Pšenda]

8 hours ago, JimmyJack said:

when rotated

Just a note - I have noticed many times here on the forum that if the object is displayed in a strange way, the cause is often its rotation (even unwanted).

[Old Bruce]

On 5/16/2022 at 11:38 PM, Texel said:

I'd like to fill a selection boundary and have the alpha channel set based on pixel coverage.

It is.

Are you wanting a clean sharp edges or a feathered edge?

16 hours ago, Texel said:

I'm talking about filling or painting inside a selection boundary. The is no coverage anti aliasing. A pixel is either colored or not.

Have you tried using a fractional pixel feather with the selection tools? This will give you a blurred edge regardless of whether or not the selection has been rotated.

[thomaso]

I haven't read/understood all the details mentioned yet, but it reminds me of this earlier issue (logged as 'afb-4917') describing pixel inconsistencies at high zoom levels respectively strongly scaled images, and not related to the force pixel alignment setting. (I can still reproduce it in v1105, although it seems to be reduced from the app version 1 year ago).

[NotMyFault]

On 5/17/2022 at 8:38 AM, Texel said:

I expect the result to look the same as if using the Rectangle shape tool and rotating it.

I think this expectation is incorrect (in the sense that Photo differs, and this is no bug).

The main cause is mixing up anti-aliasing (a method used while blending layers) and feathering selection. Both are different. Selections are never anti-aliased, but maybe feathered. Both are similar, but not identical. Only the pixel content being selected might be anti-aliased.

• You can feather a selection while you are still editing the selection, but:
• you used the rectangle selection tool (without feathering), so no hard pixel-exact edges 
• There is one important step missing between 2. and 3. you need to select the move tool, so you are no longer editing the selection.

On 5/17/2022 at 8:38 AM, Texel said:

Steps:
1. Create a small pixel layer on empty RGBA8 image (eg. 64 or 128 pixels square)
2. Create selection rectangle
3. Rotate selection rectangle 45 deg eg. by Toggle Quick Mask and rotate handle.
4. Try fill selection via Fill & Paint Brush tools.

• When you rotate the selection using the move tool, (with sharp non-feathered edges), it stays non-feathered. actually what happened is something like (a mental model, not technical)
  • the selected area is virtually copied as new layer
  • the layer is rotated
  • the layers gets blended back into the original layer (this is using anti-aliasing)
  • the copy is deleted.
  • The selection never gets anti-aliased. It stays "sharp" as it is. You can try to use the freehand selection with "shift" modifier, it does not anti-alias the selection.

In my view, the current behaviour is technically correct, and there is no strong reason why your expectation is valid. 

 

 

 

[NotMyFault]

Another way to better understand selections and rotated layers:

1. create new document of small size (256x256)
2. create pixel layer, fill in blue
3. create pixel layer, fill in green
4. select move tool
5. rotate green layer by 45 degree
6. create rectangular selection
7. use erase tool and erase selected area, in green layer
8. use erase tool and erase selected area, in blue layer
9. inspect one layer at a time
10. blue layer shows sharp edges
11. green layer shows anti-aliased edges (because of layer rotation! not because of selection)
12. to verify, un-rotate the green layer. now you see that it shows sharp pixels, no anti-aliasing.

now try the same again, but feather the selection by 2 pixels at step 6. 

 

[Texel]

9 hours ago, NotMyFault said:

Another way to better understand selections and rotated layers:

Tried your steps and observed the behavior. Typically selection has no AA and feathering does what feathering is described to do. However feathering is not the effect I need.

9 hours ago, NotMyFault said:

In my view, the current behaviour is technically correct, and there is no strong reason why your expectation is valid. 

It appears the current Affinity Paint behavior is by design. My expectation comes from 1) my need for such feature and 2) myself coming from Paint.Net which behaves the way I describe.  However, I have not used Photoshop and could not repro this in Gimp to compare what is common behavior among paint programs.

It's also possible there is another method to achieve what I want via a different interaction sequence.  As it is, I sometimes draw arbitrary shapes then translate, scale and rotate them (in my case via the selection boundary). The anti aliased edges are important for the quality I need.  I'm sure other people would  use Designer (or equivalent) and create scalable shapes instead.

I did find one workaround... If I create the shape at much higher resolution, then copy the shape as image layer, select -> rotate/scale. I can fill the high res boundary over a lower resolution pixel layer and it does anti alias the edges.

 

[NotMyFault]

To achieve the intended behavior in Photo, i suggest the following steps:

• have a pixel layer active (src)
• create rectangular selection
• select move tool
• copy
• paste (as new layer, dst)
• rotate as needed
• fill in other color
• Select layer src
• erase selected area in src

by putting the selected area on its own layer, anti-aliasing will be applied.

If you could share what result you actually want to achieve, we might find a better workflow. You are correct that using vector shapes (available in Photo, too) would be a good approach, always giving you the desired anti-aliasing.

[Texel]

24 minutes ago, NotMyFault said:

To achieve the intended behavior in Photo, i suggest the following steps:

Trying your steps, the fill step still produced no AA, however if I move your fill step before rotate, then it does have some equivalent desired result. As you say, rotating on its own layer (everything on it's own layer = an Affinity thing) does apply AA or at least a sampling filter effect on the layer.

[Texel]

I also use selection boundaries the opposite way... I cut them out.

Eg. Here's a shape where I want to create an AA cut in the shape.

I can get the desired effect using the Erase brush and holding Shift, however, I can't preview the erase line or precisely align it like I can with selections.

[NotMyFault]

33 minutes ago, Texel said:

I also use selection boundaries the opposite way... I cut them out.

Eg. Here's a shape where I want to create an AA cut in the shape.

I can get the desired effect using the Erase brush and holding Shift, however, I can't preview the erase line or precisely align it like I can with selections.

You can create a rectangle (filled with white), set blend mode to erase, and nest it to the pixel layer. This allows to get an immediate preview. Add layer fx / gaussian blur to soften the edge.

Another big advantage of this method is that you can modify the rectangle sny time later (non-destructive).

if you want the result fixed, simply rasterize.

[lacerto]

On 5/20/2022 at 8:17 AM, Texel said:

However, I have not used Photoshop and could not repro this in Gimp to compare what is common behavior among paint programs.

In PS there is in context of selection transformations a control that lets you specify the interpolation method, where you can choose e.g. Bilinear to have antialiasing or Nearest Neighbor to have hard edges. In GIMP there is a similar feature. On the other hand, when using shapes in PS, there is no easy way to avoid antialiasing (you would need to use something like posterizing adjustment or some other work around), while in Affinity you can use directly the antialiasing control or coverage map of the layer Blend Options.

EDIT: I checked ClipStudio Paint, Corel PHOTO-PAINT and Corel PAINTER and there a rotated rectangular selection will always be smoothened in subsequent fill even when having antialiasing for selection shapes (e.g. rectangular ones) turned off. Based on this it seems that aliased transformations are less common than antialiased, and PS and GIMP were the only ones that supported full control on how selections are transformed.

 

[Old Bruce]

15 hours ago, Texel said:

It's also possible there is another method to achieve what I want via a different interaction sequence.  As it is, I sometimes draw arbitrary shapes then translate, scale and rotate them (in my case via the selection boundary). The anti aliased edges are important for the quality I need.  I'm sure other people would  use Designer (or equivalent) and create scalable shapes instead.

You can use the vector tools in Photo to make the various shapes you want instead of using the selection tools. Then just use Selection > Selection from Layer, or Selection > Selection from Layer and Delete. I think this will give you the type of edge you are wanting. There will be an anti-aliased edge/boundary after filling or deleting.

Have to say that after four + days I am still not certain about the type of edge you want to achieve after moving scaling and rotating the selection.

[Texel]

12 hours ago, Old Bruce said:

You can use the vector tools in Photo to make the various shapes you want instead of using the selection tools. Then just use Selection > Selection from Layer, or Selection > Selection from Layer and Delete. I think this will give you the type of edge you are wanting. There will be an anti-aliased edge/boundary after filling or deleting.

Took a few tries but I was able to get an antialiased subtraction effect following your instruction.

[Texel]

Thanks for all your replies.

I realize my work flow is a little unusual and stems from using features that existed in Paint.Net. I was kind-of using the selection tool as a poor man's boolean shape operator over a raster grid.

I still hope that Affinity will add antialiasing (or equivalent) as an option to all related tools. Lacerto's Photoshop demo suggests it is a reasonable feature.  I now understand more about how Affinity's layer system works. I'm not sure if it makes AA harder or easier to implement. Perhaps it's merely omitted as infrequently requested by the target market.

I really wanted just one affordable commercial tool to step up from Paint.Net for a few features that bug me. However, since Affinity has a sale on, I'm going to try Designer as that may meet some of my needs better.