Something that confuses me

So I've been watching the anime, I'm on episode 6 rn, and something that's been bugging me is how there are multiple characters that Phos has apparently never interacted with before (Cinnabar and the Amethysts, maybe there's more but idk). And I just don't see how this makes any sense. There are not that many of them, they live on a pretty small island, he/she/it (do you guys give them genders?) has been around for 300 years, and like they're the only one without a job so it seems like they just have a bunch of free time and go around annoying the other gems. So I just don't really get how it's possible that they've never interacted with multiple of these other characters before.