I broadly agree. I think the fear comes from when he rightly called far-right protesters "far right" there was wide (confected) outrage over labelling "ordinary" people as right wing.