Part of the problem, other than people being jerks, is that it was in the late 80s and early 90s that trans people started to become distinct from the drag scene. If you watch Paris Is Burning, there's even a bit of a subplot about that, and somewhat later, a Dykes to Watch Out For story arc.