[Spoilers] Doesn't the existence of the one caste fundamentally undermine the core message of the game?

I loved CoS and think it is in many ways brilliant. However it bothers me a lot that the Bard caste is a literal slave society. The 'idiot' subcaste are enslaved: they cannot leave their tier of the tower (up: the monster, down: the warriors) and the moment they have the chance to do so they emphatically leave. The game simply never explains how the slave caste is enforced, but there is no realistic alternative to some sort of violence behind the scenes, perhaps the warriors.

The reason why this bothers me so much is that the game is about an inability to communicate engendering misunderstanding and lack of appreciation/respect, yet the slave subcaste literally speak the same language and are from the same culture as the Bards. Clearly their predicament is rooted in domination and violence, not misunderstanding or mistrust. Unlike the other inter-caste relations there is no epiphany or collaboration to be had: they are slaves and they can only progress by fleeing or destroying their chains.

I though the warrior-devotee rift being mended by cross-cultural appreciation (their music) was beautiful. I though the scientist-devotee relationship being created by collaboration (curing the plants) was beautiful. And then the Bards are just there like "I guess this new world is OK even though my slaves don't bring me drinks whenever I want."