Is it rude not to say b'teavon?

I'm an olah who works in a mostly Israeli office. I've been in Israel over a decade and I feel I mostly understand Israeli culture and customs, I just can't get behind saying "b'teavon" (בתאבון) or "with appetite" every time I see someone eating, especially at my office. I find it weird that I'll be eating lunch or a snack and multiple people pass by and say it, especially people I barely know. When I see them eating, I guess I'm supposed to say it back, but I just find it weird to exclaim "with appetite!" while I'm walking by to get to the bathroom or fill my water bottle or something. And then they always thank you, like thanks for noticing I am eating? If I'm polite in every other context, would I be seen as rude for not saying it?