I think this applies to the entire Affinity Suite, not just Designer, but I'm only reporting it here.
In the Swatches panel you can right-click on a color and choose "Delete Fill". Doing that generates a confirmation popup:
The message is correct when deleting from the application palette. But it is not correct for document palettes. Deleting a fill from a document palette is undoable because document-related changes are logged in the History. Also, if one is deleting a global color from a document palette, an Undo will also re-establish the color as global, and re-attach the global attribute to the affected objects.