After the war, the North decided to mostly forgive southern fighters because of this question of who their loyalty was to. (Same for most German fighters after WW II.) Calling southern fighters "traitors" is an attempt to revoke that forgiveness and restart the war.