Well I never really use a keyboard with Affinity, I just tried it on the file I had the issue with, and it doesn't crash when using a keyboard and mouse. However, now it doesn't crash using the switch and surface pen either (or any combination of pen/mouse/keyboard/controller).
I can't think of anything I did 'wrong' when it crashed (happened in multiple files on multiple occasions). I did try to save the file and then colorpick, but that didn't change anything.
I was hoping this would be a wider spread error so more people could give input, as it only happens occasionally and I can't reproduce it right now.