Time travel is real but there's a major caveat.

I've had this theory brewing in my mind for years, but can't add to it. Maybe someone else could help me, so hear me out. Time travel is real, hell we are traveling through time right now. The only difference being speed, if you can go fast enough. Time will eventually start to distort, allowing you to have your surrounding time go by faster. That's where the theory ends, time travel is more than possible if your destination is the future. Yet for the life of me, I can not figure out a way to travel backwards only forwards. It's annoying :(