As you say, the PDF specification defines PDF units as 72 'points' to 1 inch, and that is how things are measured internally.
But just because it may be technically correct, it doesn't mean that it is right from a user's perspective.
I bet you cannot find one single user who prefers to work in points, and all other programs that I have around automatically convert to the user's preferred default unit when opening a PDF.
This includes Adobe Reader and Acrobat Pro as well as other PDF readers, CorelDRAW, Illustrator, office apps and web browsers.
They all have a user defined unit that they use instead of points, and I think it would rather practical if Affinity could do the same.
Of course we can change the unit after opening a PDF but it is easy to forget and could lead to confusion as we work.
Therefore I suggest that the Affinity apps automatically convert to default document unit, or that they add a dropdown menu to the PDF options dialog, and preferably a menu that remembers 'last used'.