I think the problem is I am in the text style panel and trying to edit the swatch and the right click does not work there. It is not intuitive that you cannot edit colors from the text panel. Maybe we should keep the interactions consistent from both the panels.
Also, when I click on the fill or stroke color for the text, it always opens the color wheel instead of the Swatch Panel, which is not very useful. I always have to then click the swatch tab again. Can we do something to open the swatch by default instead of the color picker?