Precision problem with rounded pt/mm

[elk]

There is a problem with calculation between pt and mm (and others maybe; I didn’t test). Have a look at the two examples where I put in the height of a rectangle vs. calculated the size.The problem: 1pt = 0,4mm (which is wrong because of round-up). When input the height as 500pt the conversion to mm is 176,4. When input the height via calculation 1pt * 500 the result is 200mm.

[thomaso]

Obviously the number of decimals set in the application preferences > UI determines the app's accuracy not only in value fields but also for object dimensions. For instance I set mm to 2 decimals I get a more close result for the object size.

So I need to set the number of mm decimals to 3 to get it calculated AND layouted correctly.

Indeed, I would not expect that this visual UI setting is coded "by design" to influence the accuracy of objects at all for such an app-internal calculation.

[Gabe]

Hi both,

When you type in "1pt * 500" you should be getting 176.4mm with 1decimal. The only way to get the "wrong" 200mm is by typing 1pt, return, then place the cursor back in the box (now saying 0.4mm) and typing * 500. This (second) behaviour is expected, as you're technically getting the right value, as if you were to type 0.4mm * 500. 

[Joachim_L]

To be honest: Working for the first time with Affinity the handling of decimal units was confusing to me. Say you want to have a text with a font size of 10,25 pt. Due to the default (is this correct?) of ONE decimal place, the font size is now showing 10,3 pt. I have to confess, not reading the whole manual I overlooked the preferences for decimal units. Internally Affinity seems to compute the correct entered font size of 10,25 pt, but is not displaying this.

So please can anyone tell me the benefits of shortening the display of decimal units? And second e.g. a square of 10,123456 x 10,123456 px can be exactly displayed / used on which screen? Personally I changed all decimal places to 6. Just to be sure, I am working exactly.

 

[elk]

4 hours ago, Gabe said:

Hi both,

When you type in "1pt * 500" you should be getting 176.4mm with 1decimal. The only way to get the "wrong" 200mm is by typing 1pt, return, then place the cursor back in the box (now saying 0.4mm) and typing * 500. This (second) behaviour is expected, as you're technically getting the right value, as if you were to type 0.4mm * 500. 

Hm, look at the animated gif ... I didn’t return the 1pt. AD took 0,4 =1pt.

 

[thomaso]

6 hours ago, Gabe said:

The only way to get the "wrong" 200mm is by typing 1pt, return, then place the cursor back in the box (now saying 0.4mm) and typing * 500.

I would expect that internally an object dimension is entirely regardless of the currently selected number of decimals shown in the fields.

Like it does work for absolute typed values. For instance "0,1 m" results in an according object, though the value in the transform panel is given as "0 m" in case no decimals are chosen in the prefs). So, definitely the app is aware of the difference. This awareness should work for calculations, too.

That means if the value is shown to be = 0 while the object dimension obviously as ≠ 0 then a calculation shouldn't simply count with 0 but should use the real value which is known to the app according to the object on page.

So if I have an object which is obviously larger than 0 and I type * 2 then the object should get scaled by 2 and calculate with the object size, not with any cropped value in the field caused by any current custom number of decimals. The fact that the object scales but the value doesn't show a difference in such a case is related to the selected number of decimals only, it is not a task to the app to ignore anything smaller than 1. So, unless I did not type 0 and I place the cursor behind the current value then the app should not calculate with 0 but use the true value from its memory: the same which is used to render the object on page with its true not with a rounded dimension.

It reminds me to an Affinity ruler oddity, which number of subdivisions varies with the zoom level – and may create quite strange situations this way, see above for instance "0,1875", there are at this zoom level obviously no metric subdivisions even though it is a metric unit "mm".

[walt.farrell]

If I have a Designer document set for mm, and I make an object 1pt in width, the Transform panel shows .352778 mm, rounded to 6 decimal places for display.

If I set the object size to 1pt * 100, the panel then shows 35.277778 mm. That's the same value I get if I set it to 100pt.

Setting it to 1pt*1000 I get 352.777778 mm.

So, on Windows, it seems to be behaving properly, and the calculations are done with full internal precision, and rounded only for final display.

[Old Bruce]

7 hours ago, elk said:

Hm, look at the animated gif ... I didn’t return the 1pt. AD took 0,4 =1pt.

 

Do you have any decimals set for the display in your Preferences?

[thomaso]

7 hours ago, elk said:

Hm, look at the animated gif ... I didn’t return the 1pt. AD took 0,4 =1pt.

I would expect this occurrence in the value field when you have set decimals to 0. – It would be interesting to see additionally what object size the app creates on page with this same setting and calculation.

2 hours ago, walt.farrell said:

So, on Windows, it seems to be behaving properly, and the calculations are done with full internal precision, and rounded only for final display.

How do you know? It seems you have set 6 decimals. What happens if you set to 2 or 0 decimals? Then the value shown in the field will be different because rounded. – But does the object on page get a different size, too?

I'd say it must not be different but always should internal calculate accurate. Otherwise a switch of the ruler's unit or the number of decimals for value fields would have to cause a change of the layout because its dimensions get rounded according to its value in the field.

[Gabe]

12 hours ago, thomaso said:

So if I have an object which is obviously larger than 0 and I type * 2 then the object should get scaled by 2 and calculate with the object size, not with any cropped value in the field caused by any current custom number of decimals. The fact that the object scales but the value doesn't show a difference in such a case is related to the selected number of decimals only, it is not a task to the app to ignore anything smaller than 1. So, unless I did not type 0 and I place the cursor behind the current value then the app should not calculate with 0 but use the true value from its memory: the same which is used to render the object on page with its true not with a rounded dimension.

Not exactly. If you have 0 decimals, and an object 1.4m, it will show 1m (expected)

Now, when you place the cursor inside the box and type * 2 for example, it's giving you the result of what's typed in the box, which is "1m * 2". Taking into consideration internal values for TEXT would be wrong. When one types in an expression, they will get the result of it. If you wanted to use the internal value, you would have to delete what's in the box and type "*=2" or "*2". 

[elk]

26 minutes ago, Gabe said:

Not exactly. If you have 0 decimals, and an object 1.4m, it will show 1m (expected)

Now, when you place the cursor inside the box and type * 2 for example, it's giving you the result of what's typed in the box, which is "1m * 2". Taking into consideration internal values for TEXT would be wrong. When one types in an expression, they will get the result of it. If you wanted to use the internal value, you would have to delete what's in the box and type "*=2" or "*2". 

This one is interesting! I didn’t ever think of only putting in a factor. So – when working with calculated fractions of elements this way seems obviously the best – independent of the displayed decimals.

[thomaso]

8 minutes ago, Gabe said:

 If you wanted to use the internal value, you would have to delete what's in the box and type "*=2" or "*2". 

Wow, thank you Gabe!  Sorry, I did not even think of this possibility!

This still can be shortened (= without need to delete the current, rounded value) if I just select/highlight the entry with a single click inside the field and simply type *2 directly. Very smart + handy! 😍

One more question about the visual experience:

When I click the label the field's boarder gets highlighted and I can start typing – but can not delete in this state: 

When I click the field's content the entire content gets highlighted and I can start typing incl. (auto-)deleting the current content: 

Is this the way it's designed to work? I ask because I experience a horizontal tab order in the transform panel (X > W > Y > H > R > S) while other users report the order vertically (X > Y, W > H, ...) which appears more useful. So I wonder if my highlight experience is as designed (or, e.g. if the 'delete' key should also work when clicking the label only).

[elk]

11 hours ago, Old Bruce said:

Do you have any decimals set for the display in your Preferences?

I fiddled too much – so this was set to 0 decimals, I’m afraid. Sorry for this confusion. The internal calculation is correct (see new anim gif). But Gabe makes it really clear.

[thomaso]

1 hour ago, Gabe said:

If you have 0 decimals, and an object 1.4m, it will show 1m (expected)

Now, when you place the cursor inside the box and type * 2 for example, it's giving you the result of what's typed in the box, which is "1m * 2". Taking into consideration internal values for TEXT would be wrong. When one types in an expression, they will get the result of it. If you wanted to use the internal value, you would have to delete what's in the box and type "*=2" or "*2". 

Wouldn't this valuable hint deserve to be mentioned in the Affinity Help at "Expressions for field input" near these blue boxes?
For instance " ✏️ Delete an existing entry to avoid calculation with rounded values" ...?

https://affinity.help/publisher/English.lproj/index.html?page=pages/Workspace/expressions.html?title=Expressions for field input

@Pauls, @Patrick Connor