Hello @dominik,
I just got back to this. I can open your converted file but it still caused publisher to hang when I selected a chunk of it.
It's interesting to see how differently the file seems to behave on different systems. I can open the file (original) in about 4 seconds and see it just fine. The converted file takes about 6 seconds.
In any of the pdf's I can select a "bunch" of items from the Layers panel, then copy and paste into another document works fine (but if I select too many it hangs).
As others have pointed out, publisher is not supposed to be a pdf editor, but being lazy, it's nice to be able to open pdf's (or other formats) and grab the pieces I need to composite in my "final" documents without a lot of switching back and forth. Affinity generally works pretty well for this and maybe the file I happen to borrowing from is just messed up.
The lock-up behavior I would still consider to be a bug, so hopefully that may get taken care of in future releases. In the meantime, thanks to all for suggested work-arounds and comments