What freaked him out and have you discussed why it freaked him out? A lot of guys indulge the fantasy but need time to understand that the reality might not be all he thought it would be. Two ways he will respond IMO, either he freaks out and realises he doesn't want it to happen or he just needs time to figure out how much he really does want it without risking losing you, first scenario you can't do much about (unless you decide you are going to do it regardless) second scenario is that you both have to convince each other that you want it and its not going to damage your relationship irreparably... Just my 2 cents worth..