It's just a fact that neurotypical people are the majority and don't think or speak literally like you would want. That means it's up to you to be specific when you want feedback that neurotypical people might often find objectionable and just wrong to give in a certain way. It's not fair.