Why didn't Joel and Ellie and Tommy go together?

Why was it such a big deal that Tommy take Ellie instead of Joel? Why couldn't they all just go like as a group of 3?

This is from episode 6 btw