Cons and obstacles of owning and flying Drones? (First time buyer)

What's some obstacles/struggles you deal with when operating Drones?

I feel like Drone flying is generally unaccepted nowadays by the general public, government and law enforcement and I generally am worried I'll get the cops called on me for operating a drone.

Are people typically OK with someone flying a drone? Or do you get threats or anything like that? Am I worrying too much?