If he was really uncomfortably large, finding someone less endowed shouldn't be very difficult.
However, if it wasn't terribly painful, having him take some extra time masturbating and slowly stretching you out might be all you need, if you are considering him as a "steady". Since your hubby likes to watch, you might have him help out before your "date" arrives by starting the process of gently stretching you open with his fingers and a lot of lube. You might try a dildo or two as well.
I need to do this for my Wife before we have intercourse as well, so that it isn't painful for her when I enter her. It's also a lot of fun to do by itself.