Yeah, I forgot about the Surface Studio, but yes that's also a use case.
Anyway, I have good news! One of the changes in beta 1.7.0.284 was: "Added 'Touch for Gestures Only' option in Preferences"
I've just tested the said option and it works perfectly! No more stray marks!
Thank you Affinity team! You've just made the workflow of all illustrators using Designer (and Photo) much easier and faster! This issue can now be marked as closed and archived away. Thanks again to everyone!
I am using a Surface Studio 2 and am constantly removing elements drawn unintentionally when I am trying to move or zoom the document. This is a workflow killer. Being able to disable gestures from altering the canvas content would be a huge productivity improvement.