Do references ruin stories for you?

I seem to be running into a lot this of late- where characters will make those pop culture references, like explicitly. "Like in the movie The Thing" - "have you ever read lovecraft?"

I get that in real life, casual conversation, people will make references to things they like. But when you are crafting a story- what this feels like, is that you couldn't convey things without cribbing on a better known work- if this is part of your characters discussing your main plot.

Sure, someone talking about their favorite movie if it's not relevant to your story is okay, but it feels like a lazy choice when that's the only way for someone to explain something important with in it.