We, we, we!

Anyone else get annoyed at all the "we" references?
My SO sometimes uses "we" when talking about his past, as in "when we bought our first house" or "we always had strict bedtimes with the kids". Logicially I know that everyone has a "we" in their past lives. My partner and I are divorced from long-term marriages and we each have our own bio kids, so I am a BM and an SM, but when I am referencing something that happened during my prior marriage, unless my ex was relevant to the story, I just say "I" ("when I bought my first house"). My shift in wording didn't happen overnight, as I was previously married over 20 yrs and very used to referring to my former spouse and I as a "we", but I've been divorced for 5 years and with my partner for 3 yrs, so now it just doesn't feel natural to say "we" anymore in that sense.

Partner's ex can be HCBM and there were definite boundary issues that we had to work through at the beginning of our relationship (she leaned on him for emotional support and they were still playing happy family kind of stuff), so maybe there's still some residual irritation on my end about that, and I have my own emotional baggage from my prior marriage (infidelity, runaway husband), so maybe that plays into It.

On one hand, I know it is such a small thing. But it grates on me. I've talked to him about it a few times. I've explained that it's not really the reminder of our pasts that bothers me, but that in my mind, there should only be one partnership (romantic/spousal type partnership, that is) and the only "we" partnership right now is US. I'd be much less bothered by him saying "When ex-wife and I bought a house" than all the "we's." He's gotten better about it after I mentioned it, but he still does it probably once a week and I can't help but think, 'Is it really THAT hard for you to make this minor adjustment?! If I can do it after being married to someone from 20 years, can't you?' Especially when he knows that it bugs me. I can deal with knowing the exes will forever be in our lives when it comes to co-parenting, but do I forever have to have the shadow of another "we" hovering in basic conversations about our pasts? There are so many other ways that we have to be accommodating and picking our battles when it comes to step parenting, dealing with co-parents, etc., I guess I just feel like this is my super sensitive pet peeve.