No, never. If I am having sex, it is because I want it, and if I want it I will have an orgasm. Just because a dick is smaller does not mean you won't have an orgasm. A bigger dick may give you a better or different orgasm but not to say a smaller one cannot. I have had big and small and everything in between.
And if I am not into it, you ain't getting into "it". I will jerk you off but that's it.