They are already gonna lose them. Nobody that watches Colbert is going to watch anybody they can scrape up to replace him. They'd have to attract a whole new audience.