I have the same sensations on my late 2012 iMac using Windows 10 (i7-3770 / 16GB / 2GB GTX 680MX).
Only Windows is installed on my machine. It's a native EFI install, I don't use Bootcamp nor Apple's drivers.
I purchased Affinity Photo for Windows to support Affinity (and to prepare my switch back) and I was, just like you, expecting far better performances than the OS X macOS version.
I really hope the coming updates will significantly improve Affinity Photo for Windows smoothness.
It would be interesting to have a script to test machines and compare both OS versions.