I think I'm experiencing the same infuriating issue (bug?) with my document having insanely high file sizes when exported. I'm using the latest version from the Mac App Store. I think it's a font problem, but I think it's one that Affinity can fix on their end.
As a clean test, I created a brand new letter-sized document consisting of just a single large text frame filled with 8-pt type. The font for my project is EB Garamond Bold (a nice free and open font).
When I export to PDF using the native macOS print dialog, the output file looks perfect. All text is fully selectable in Preview.app and perfectly sharp even at maximum zoom. The file size is 37 KB.
When I use the built-in Affinity export dialog, the output file ALSO looks perfect. However, the text is NOT selectable in Preview.app anymore. It's all artwork now. The file size is a whopping 8 MB !! This leads me to believe that all the text has been converted to curves (though I don't have an easy way to test that).
I tried it with a bunch of other fonts—system fonts, commercial fonts, free fonts, and the problem does NOT occur on any of them. This explains why this "bug" isn't very widespread. In fact, I even tried with with EB Garamond REGULAR and the problem does NOT occur. That's why I'm hesitant to even call it a bug on Affinity's part. Maybe it's a bug in the font file. Unfortunately, this is the font that I'm stuck using for this particular project. And I don't think the bug is insurmountable, since we don't see the problem occur on the native macOS PDF exporter, and we don't see it in other software.
Hopefully this is something that can be fixed in a future update. But honestly, once I'm done with this project, I'll probably never notice either way. Just wanted to do my part to contribute to the thread.
EDIT: I fixed the problem by deleting the font from my system and downloading a new, slightly different version. I had to go through my document and tweak a couple things because the spacing had been changed ever so slightly, but all is good now. In the future, I'll know to test my fonts to make sure they embed correctly before using them in a project.