Selection Refine... conundrum

[pipet]

I use selections here and there, but in a haphazard kind of way, as I've been hampered by some odd behavior when refining those selections.

I first make a rough selection using e.g. the Selection Brush Tool, then I go into the 'Refine...' dialog and start painting the edges to refine. At this point the selection usually looks very much to my satisfaction, especially when checking with the 'Matte Black' preview. But whenever I then click 'Apply' in the refine dialog, the resulting selection is all messed up and a far cry from what I saw in the refinement dialog. What' am I missing?

[GarryP]

Can you show us an example of what you mean by “all messed up”?
It’s difficult to diagnose a visual problem when you can’t see it.
A screenshot of the original, and one of the matte (while refining the selection), and one of the result may be useful.

[pipet]

Happy to oblige. Below an example from an insect I shot (partial detail).

So here's the rough selection with the selection brush:

 

This is the selection after refinement and with the black matte review (including all the fine hair detail as intended):

 

But after clicking apply, I see this, which still includes the fine hair detail, but obliterates some parts of the leg, which was included in the refined selection above:

 

 

 

[GarryP]

Thanks for supplying the images.
Hopefully one of the selection experts will see this and be able to advise.
(The problem may simply be down to the settings values you are using – Smooth, Ramp, etc. – but I could only guess while an expert will be able to steer you in the right direction much more quickly.)

[NotMyFault]

A general tip when selecting on low contrast images like this :

just add an brightness/contrast layer and push its settings quite a bit to increase contrast as far as possible. 
you may use HSL to boost colors, too.
then merge visible. 
the resulting layer can then be used for selection. 
after having a good selection, simply hide or delete - then apply the selection to the original layer. 
 

using this helping layer will give you much more visible feedback of you selection. 

[NotMyFault]

Another option in your case: the background seems totally black. You may use selection by color of the background, then invert selection. Could provide better results. 

[NotMyFault]

Another tip: before using color-based selections, check if your document is at least 16bit color mode. In 8 bit modes, selection can become erratic (all or nothing)