I'm not really sure it's guilt by association when people go on local Facebook groups and make it clear exactly why they're doing it. And also say "It was me who vandalised the mini roundabout and I'm doing more" and are praised by huge numbers of local people.
You're implying that the people who support it are the equivalent of Britain First or Tommy Robinson, assuming it must be exclusionary by nature. A more charitable reading would be to see it as an expression of English identity, a demand to be heard for what it is, not for what it is presumed to be.
The issue at the heart of what im getting at is that reflexive reaction progressives have to the flag. It is terrible for unity. In our secular nation, what else can we align under whatever race or religion, if not the flag? Why abandon it and allow its power to be abused by the far right?