How can a gay man not feel anxious around straight men?

As a gay man, I always feel anxious around straight men to point where I feel like can’t be myself around them out of fear that I have to deal with homophobia from them. I always feel like I have to make my voice deeper than it already is and make sure that my mannerisms don’t come across as “obviously gay”. I don’t see myself as the most feminine or flamboyant gay man in the world, but at the same time, I always felt like I wasn’t as masculine as most straight men. I was never into sports or any other stereotypically masculine things growing up. Also, I was bullied by some straight boys for being perceived as gay in both middle and high school.