Feeling stupid

I often have trouble understanding what people say. Especially fast talking or when it's a topic new to me. And even worse when it contains numbers or dates or people calculating loudly to demonstrate how easy something is. When something is presented visually, I get it instantly. Even when it's complex and I am surprised when people find visuals more complicated then text, when even someone "stupid" like me gets it immediately. It's like those two skills are switched in my brain. Wich can make social situations awkward or misunderstandings happen.

Do you have any idea what's going on? I've been talking about it with a doctor. He kind of brushed it away. I shouldn't worry. People are just different. Some are visual. Others not. But often it really is annoying or producing anxiety of appearing stupid in front of other people. I learned to live with it. A lot I do is visual now. But an explanation would be nice.