Brownlee from the outset of the debate decided he was going to come down hard on provocative speeches, and had already kicked a Labour speaker out. So he wasn't at this point specifically biased against Swarbrick; he was just managing the debate poorly, with unreasonable standards
Now that happens, fine, but perhaps when you're kicking people out on a hair trigger to manage the debate, this isn't the time to also come in with innovative interpretations of your power. Just let bygones be bygones, jeepers. Worse things have been said in parliament about less important issues.