I mean "damage" in the sense of the ability to pair-bond and build deep, trustful, and intimate relationships.

This doesn't mean it makes someone a "bad person" - it just means meaningful intimate bonding becomes harder. As far as I know, there are many studies and countless anecdotes backing this idea.