What are your thoughts on player only spells?

For context, I was playing a game with a DM that worked as a kind of introductory D&D 5e game in our community. I'm an experienced player who serves as a mentor for newer ones at the table.
Any time we had a character with an AC of 18 or more, the DM would have the monsters we were fighting hit the character with Banish (even though we were level 3). Because it lasts a minute, players would basically have no choice but to just sit and watch everyone else play the game. Afterwards, I brought it up to the DM and pointed out that we may lose new players who get bored because he magically puts them in time out for having a hard to hit character. His argument is that it's a fair spell because their high AC throws off the balance of the fight and that the players should target the spell caster to try to bring back their ally. Since it's in PHB, it's free for the bad guys to use. Personally, I think it's kind of a dick move to use it on low level characters who can't defend against such a comparatively powerful spell. So, what are your thoughts? Are there some spells that DMs shouldn't use against players? What sort of limits do you guys put at your own table?

Update: Thanks for the award!

I do want to clarify, this was not a one-time thing. The DM was, and still is, one of my best friends, and anything he does is less likely malicious and more likely anxiety or ignorance.

He struggles a LOT with game balance, and though I play with him in many games, I'm definitely not a consistent player in the games he runs anymore.

I'm a pretty experienced D&D player and game master, since tail of 3.5e to present day, while he's been DMing (and playing) maybe 3 years at this point.

If I'm in an introductory game with a new DM, I usually let (imho) bad DMing slide until after the session. Then ask the DM if they want feedback for their session. He does want to improve, but I think he sometimes forgets that new players don't know what to do.

When he banishes me for third time that session, I do remind the players (politely) that the spell caster should be targeted, but being new players, they rarely do that.

Lastly, some in-game context: I was a Grung Artificer, Battle Smith (planning to multiclass fighter). We did roll for stats, so I had a high Dexterity and used my infusions to buff my AC up to a 19.

Story was we were supposed to meet a mage to charter a ship to an island where the main plot was located. Party wanted to prep first by buying equipment we may need, only for us after to find our contact dead in an outhouse.

When searching for clues, the DM indicated we found a tunnel leading into a passage hidden under the outhouse, so we naturally went inside. As soon as we went in, we rounded a corner to see 4 heavily armored cultists, two men in wizards robes and a red dragon wyrmling, (no warning).

I rolled first and went to order my steel defender to go for the spell caster, but DM told me since I didn't describe my steel defender climbing into the tunnel he was still at the entrance and that a cultist had kidnapped him and dismantled him. Note: We were 60 feet from the entrance and no dice were rolled. My DM notoriously hates summons or companies (another issue for another post). We were also told that we couldn't escape as the entrance had been locked behind us. So I decided to fight myself and went up to the dragon because narrow tunnel, he couldn't breath weapon without hitting his friends.

That's when random no-name spell caster used Banish on me. I didn't roll high on save, so I was banished. New players did break me out once, but then spell caster two also banished me and they didn't want to focus him for my sake for fear I'd just be banished again. We had to stop mid combat, at which point I pulled my friend aside and had the aforementioned conversation with him regarding the use of Banishment on new players.

I later found out he was spitballing the whole encounter and he said he didn't know what to say when we rolled high on investigation for the outhouse for clues as we weren't supposed to find anything about who killed our contact.

I ended up surrendering the point and changed my character to appease him. Bad move on my part, but I'm not one for interpersonal conflict.