Well, to be fair you can be athletic and have man-boobs, I'm sure there are many football players who have them. What about wrestlers, and weightlifters?
I am pretty sure the guys describing themselves as athletic are not former division one athletes. When used to describe body type athletic tends to mean well toned with low body fat.