I keep getting called by those scam numbers, should i worry?

I answered once by mistake so im worried specially that i saw another post today about this and the comments were like go call the police