'Literary fiction' is an annoying label

And a completely meaningless one. It's a marketing tool that has no substance behind it. Even the wikipedia 'definition' is just a pile of disparate associations.

No author of 'literary fiction' thinks they are one. Martin Amis, Ian McEwan, Ruth Ozeki, Cormac McCarthy, whoever, you name the person. What they think about is the great literary tradition they draw from. And this might be the only unifying trait among these very individual-thinking authors: being grounded in literary tradition. As opposed to pop-culture tropes (not a jab at genre fiction, mind you).

An 'introspective', 'character-based', 'philosophical' book is best described with exactly those terms and not as 'literary fiction'.