I've noticed this issue too. The issue seems to be "drag distance". If I keep the dragging in a small radius around the point, it will never drag. As soon as I move outside of the boundary, the drag starts. This makes it impossible to do fine movements (which is what I use this tool for - small adjustments). I see the same behavior when using regular touch - like 5 points distance of panning is needed to start the drag.