AITA for not wanting women in the men's restroom?

I'm a truck driver and I was stopped at a pilot. I just woke up, went inside and went to the restroom to take a #2. All the stalls were full except one. Well, as I'm finishing up, I hear a guy walk in talking and then hear women responding. I thought I was hearing stuff or maybe they were trans or something? Idk.

Anyways, I got done with my business and walked out of the stall just to find two women there (employees) and a man (the manager) standing there talking. This honestly made me mad. I immediately asked, calmly, why there were women in the men's restroom? (At this point, another guy left a stall and immediately booked it to the exit. You could tell he felt uncomfortable.) The manager said he wanted to show these ladies the urinals. Something about them being messed up or something.

I said, "No. That's not how this works. Women are not supposed to be in here. There are men in here."

Manager, "What's the issue? The stall doors are closed."

Me, "That doesn't matter, guy. By that logic, why don't you go into the women's restroom?"

Manager, "...."

Me, "In fact, what would happen if I go into the women's restroom?"

Manager, "..."

At this point, I knew he knew what I was getting at, but he was still looking at me like it wasn't a big deal. Finally, he told the women to leave.

I responded to him, "That was really dumb man."

He shot back with sarcasm, "Yeah, I'm just stupid."

Me, "You're God D@mn right! Do you see the world we live in? You know how easy it is for them to get us in trouble? This was really dumb man!"

I may have been a bit on the angry side. But I feel that the men's restroom is FOR men. It's our area. Women have their area and us men would get into deep shit caught in there. Which is another thing. I HATE double standards. Idk. Am I an asshole or Karen for doing that?