I'm having similar problems with anchors in Affinity Publisher. I'm trying to set up a manually generated Table of Contents (manual, because it needs extensive customisation). But it's incredibly frustrating.
Firstly, Publisher puts anchors in automatically where I haven't told it to, such as on sub-headings with classes I've created (so not inbuilt heading levels). It also puts unwanted anchors in running chapter titles generated from section names, filling in text boxes from the master spread. This results in a many-page article having many identically-named anchors.
Secondly, the Anchors panel is almost useless. Trying to delete anchors in it is very frustrating because trying to put the pointer on the trash can icon is so close to the bottom right of the panel that the Mac constantly thinks I'm trying to resize the panel. Anchors in the running chapter titles can't be deleted at all. Also, trying to turn off the "Export as PDF bookmark" setting on individual anchors doesn't appear to work from this panel. And surely it should be possible to EDIT the anchors from this panel?
Thirdly, something weird sometimes happens with hyperlinks to anchors. I've hyperlinked items in the contents pages to the appropriate anchors, but sometimes this doesn't "stick" and I get pre-flight errors saying the hyperlink is invalid. No matter how many times I delete the hyperlink and relink to the internal anchor (which DOES exist), it doesn't stick and ends up being hyperlinked randomly to some other anchor, which is invalid (I think it's one of the ones in the running chapter headings, perhaps because these fields are set up in the master pages?).
It's all very disappointing.