I accidentally embedded a large image once, and even though I removed it, the file size never went down. My file is now permanently 100x what it should be. Searching the V1 board, I found out that Affinity files keep all the old data, just adding changes, until it reaches a certain point, at which time it does a clean-up. I don't think I'll ever reach that point, as I link all my images now.
There should really be an option somewhere to clean up old data in your file (garbage collect). It's so frustrating having to live with large files that don't get cleaned up because the original mistake was so egregious. My file is currently 200MB, with no embedded images at all.