It's derived from it. If it it points to nothing other than the skin color, which it does, then the defensiveness against it is the exact thing that suggests being black is something you shouldn't remind people of. You unknowingly (probably) suggest that being black is something people can be teased with.
They should make an international day for using that word to take pride in what it refers to (the color) and only that will defuse its currently (and centuries old) perceived meaning. Then make it a week instead of a day..
I have to guess your IQ is average or slightly below average. Take it from me. You can't even make an argument.