I am so accustomed with CTRL+Click with other tools, that yesterday when I was trying to do this thing in affinity, it took me half an hour to to test ALT, SHIFT, CTRL and in different combinations, with no success.
Additionally, I tried to search for documentation, and I found this(which explicitly states that flood selection tool does not support this): Add to selection (Marquee/Freehand selection tools only) CTRL+ALT-drag
which is specified here: https://affinity.help/photo/en-US.lproj/index.html?page=pages/Workspace/shortcuts.html?title=Keyboard shortcuts
I had no way to test what you said about LMB and RMB yet, will do ASAP, hopefully it exists, but nothing is specified about this combination. Do you see it there on the list, for selection, because I don't