Well first off, being a man is not an immutable characteristic, because transgender people exist.
Second, the difference between the two groups you bring up isn’t relevant in this comparison, because pointing out the differences between them doesn’t negate the similarities. Both are groups with an inordinate amount of power (physical or legal) over any outgroup and are supported by cultural norms that allow them to exercise that power largely without repercussions. Both groups are also protected from consequences by others within the group, regardless if the others agree with their questionable decisions. And most importantly, both groups are human, meaning the individuals vary widely within the group along the moral spectrum. Even if most within the group are good, bad actors will always exist and there’s no way to know which one you’re dealing with at face value.
Even if it’s not a perfect comparison, it’s apt enough to support my point.
I’m American and I always get a chuckle from the adoration that people have over raccoons as well. I guess they’re cute but they’re also a menace, there’s a reason we call them “trash pandas”.
But I also went to Spain several years back and saw my first hedgehog. And it was even in a hedge! I took probably two dozen photos and the locals thought I was crazy. So I get it.