Thanks for the reply Old Bruce!
I am on Windows 10 Pro (2020 Edition).
You are correct that I tried using Affinity with IDML files at first, however I quickly switched over to using AFPUB files. Both file types are slow for me. The AFPUB files are comparatively faster than IDML files, however they are still excruciatingly slow. When I was comparing the software, I actually built the same book in both InDesign and Publisher, and while there are a lot of things I prefered about Publisher's interface, the performance was just incomparable. Publisher was terrible to the point of being unusable, while InDesign was extremely fast and responsive.
In regards to Save History, I am not using it. In regards to the file sizes, it's crazy how random it is. While working on a file, it started at 150 MB. I saved it and it went down to 80 MB. After working a bit later, it was 1.5 GB. I did a "Save As" and made a copy of the fifile just to see, and that was 1.2 GB. It's all over the place.
For the resources, nothing is embedded. The linked files are large images (PSD) and there are a lot of them, so my guess is that the program is actually embedding them when it says it's linking them, but I can't confirm that. I'm not sure if there is a similar setting in InDesign, but if there is I don't use it.