I like the idea of controlling him....but I don't think I am mean enough to do this to him
If it is something he is into, it isn't being mean at all, many men enjoy the tease and denial aspects of chastity.
So, as long as you approach it as a game wherein you do everything you can think of to keep him stimulated and desiring you, he will respond by doing everything he can to try to get you to allow him to have an orgasm and even that you get to control by dictating how he can do it.
The thing you need to avoid is permitting him to whine about being released. You want him to beg appropriately without whining and to perform as you wish to earn his release. Think of the benefits: he does the housework, treats you to dinner, massages your feet, performs oral sex whenever you wish without reciprocation, etc.
You in turn, dress provocatively, allow him to see you but not touch (except when he gives you massages), touch him in sexually teasing ways, have him make love to you without using his penis, allow him to watch and/or serve you and your lover. Do whatever amuses you to keep him horny and willing to do anything to cum.
Then, when you decide he has earned an orgasm, you can be nice and allow him to fuck you, give him a blowjob or hand-job or if you would rather be nasty: tie him up so he can't move and then edge him over and over and give him a "ruined orgasm" by bringing him right to the point that he begins to cum and then immediately and completely stop all of the stimulation you were giving him.
Many women find it quite amusing to hear her man begging her to keep stroking or sucking him while watching his penis helplessly twitching and spurting out it's cum knowing that even though he is shooting his load, his orgasm has been ruined and he will still be extremely horny but unable to achieve an orgasm again for quite a while.
Of course, if you choose to not grant him an orgasm for whatever reason, you can always "milk" him by stimulating his prostate gland causing him to expel his semen which will keep his "plumbing" in good working order, but deny him the pleasure of an orgasm until you decide he may enjoy one.
Many women practice this as their normal regimen in order to keep their husbands in that highly aroused state where they are eager to please, but make sure his body avoids the "use it or lose it" syndrome by giving it the work out it needs without providing the pleasure normally associated with it.
Oddly, the men who enjoy this sort of play find themselves torn between desiring the orgasm they crave and hating it when they receive it because it ends their arousal and desire for it until they are denied for a period of time again. So keeping your husband in chastity may actually be providing him with a happier sexual life than otherwise. Beside, during the refractory period after having an orgasm, he will probably be much less submissive and somewhat lethargic until the desire for sex begins to peak his interest again.
Again, it's different strokes for different folks. The key to all is communication. Have fun.