As far as married couples go, it all depends on how each individual spouse feels. If both feel it is exciting and adds excitement to the marital relationship, absolutely not, and its really no one else's business.
As far as developing "trust" in a marriage, IF the male part was to desire having occasional sexual affairs with other females, would the female spouse agree to his doing that? What I've found is ... in most cases NO. Married women struggle with this 'cuckold' desire of men anyways; it makes no sense to many of them as to why their spouses would want them having sex with other men. So, have they really developed a trusting relationship, often not.
My wife doesn't have an issue with me having sex with other females as long as its in her presence. She would never allow me to go out on dates or have an overnight affair. Its not that she doesn't trust me; she says she doesn't trust other females. (hoooohummmm)