XM4 - Auto pause when taking headphones off

Just got a pair of XM4s but I’m having an issue with the auto pause feature. I have checked the settings in the Sony app and tested a bunch of times, when playing music and I take off my headphones, music still continues to play.

I have seen review videos online where the auto pause feature pauses music almost instantly when you take off the headphones. For whatever reason I can’t get this to work.

Is anyone else having the same issue? Any other suggestions are welcome.