If you have a modicum of reputation, if you are known, however minor, for anything, there will be people who don’t know you who don’t like you.
We make judgments about others based on information to hand.
We make those judgments from either stated facts or something synthesized from those facts.
The judgments aren’t always sound, we know they aren’t sound, but we will make them anyway.
There are people in this world who don’t know you and they don’t like you.
Get used to it.