Will I regret losing my virginity to a hookup?

I (M21) have never done anything with a guy, I really want to but am scared I’ll regret it. I’ve had a horrible experience growing up with my sexuality and just want one good thing to come out of it. I met up with someone today, he was really sweet and we went out for lunch. Then back to his place where we watched a film and cuddled for a bit. I was attracted to him and hard while we cuddled. He said we could try kissing but I couldn’t even bring myself to do that, what is wrong with me? He was very understanding so we just finished the movie and I left. I feel embarrassed that I wasted his time, am I being silly? Is your first time really a big deal?