Hi guys. I just wanted to chime in, having the same kind of performance trouble on two very powerful workstations (min specs: i7, 3,5 Ghz, 16 GB DDR3 2133 Mhz, NVME Drive for the program and swap, GeForce 970).
I have observed that at the times of worst lagging and stutter Affinity Photo takes almost all the free physical memory (so on the more powerful machine 22 GB) while working on a 24Mpix 16 bit raw file. It is surely big, but not that big.
I do have a feeling that it was better while i was trial using the program (also 1.5) some 4 months ago. Now I'm not sure whether that has to do with some recent windows 10 updates, different camera (jumped from Canon 6D L-raw to 5D mk IV M-raw, so file size is more or less the same), or just pure luck (or even faded memory thing).
I sure can't wait to test a stable 1.6 release.