Aren’t the first few round table banishments meaningless?

Surely, any traitor that gets banished in the first half of the series will be replaced by another traitor anyway? Is this not a bit of a flaw in the system? Does it really matter who is being banished? Might as well just get rid of those you find most annoying?