Thanks, I just tried some tests with the rasterizing and the one that worked the best was "Rasterize & Trim" after creating the mask, not before. Then, after rasterize and trim I could add layer fx that only effected the masked area. You can see from the screenshot how the selected layer has a bounding box around the image only. Thanks for the tips!
So again, for anyone else having this issue: Rasterise & Trim after creating the mask, then add layer fx to the trimmed layer.
jdrux got a reaction from walt.farrell in Document edge and layer FX don't mask all the way to edge.