One really neat feature of the Affinity Suite is that each of the three programs can open any of the others' files. Try it! Open a Designer document in Photo then vice versa. It's really neat & powerful.
The difference though is the interface and tools each program is set up for.
Designer is best if you are working with vector artwork and typography. If your cards are mostly line art rather than photographs, I would use this. If you're familiar with Adobe, it's like Illustrator.
Photo is a photo editor. Use this if your cards are mostly photographic, if you you need to do fine photo editing. It's similar to Adobe Photoshop.
Publisher is for laying out and typesetting longform documents, like books or pamphlets. But it may be useful if you want to set up a single template and store all your cards as spreads in a single document. It's similar to Adobe InDesign.