Among many isolated white communities this is the case. In Zimbabwe, for example, the local white Rhodie community, tended to ostracise members of their community who associated sexually with black Africans. The white expat community had no such social restraint.
In the UK there are many parts of the country were there are very few coloured people. In the current town I live in it is 99.5% black and the black people who are here are more likely to be american serviceman than local black people. In many parts of the UK similar communties are the normal state of affairs, and in such places invlovement with black people would raise eyebrows and start gossip.
It is in some cities where there remains social tension that the taboo's remain among some in the community. Every white community is likely to have some rascist elements. However, that works both ways. Some Asian communities frown strongly on relationships with whites, though they do not always frown on the sexual exploitation of whites by members of the community.
When I travelled through rural westen USA I came across very few black people, and though their absence meant there were few signs of prejudice. I gained the impression that while friendly as a whole some of these communities were quite closed and suspected a sexually aggressive black alpha male would have been as welcome as a rattlesnake (at least by the men
)