What causes our fear of being perceived?

This is a question I find myself mulling over fairly frequently. I know it’s an autistic thing but I have this “chicken and egg” discussion in my head about whether it’s something we’re born with, or something we develop because of negative interactions.

I can spend a long time ruminating over all the negative interactions I’ve had with people that would lead me to never wanting to be noticed by a human being ever again. But then I also had this pathological fear of being perceived as a small child (4-5 years) and could I have had that many negative experiences at that age?