the thing is, if your wife does not fantasise about sex with other men, then there is never a good time AND anytime is a good time. and dont fuss about her fantasy and your fantasy. just focus on 1 fantasy. hers, yours, whatever.
in short, there is nothing stopping you from introducing fantasy...