We can never go backwards in time though.

How about if we travel 12 months into the future?

You're assuming that time is tied to space relevant to the sun. Wouldn't it be more likely if not tied to the planet then but the entire universe. The solar system is moving as well as the galaxy in relation to its spot in the universe. In your diagram that would mean that you can still travel freely in time but only at 365 day intervals (sure you'd need to account for leap year). If you can manipulate your point in time you can most likely manipulate your point in space to make the entire discussion irrelevant as every body in space seems to move at a predictable rate.

Actually, more like this (that's how many miles the sun travels around the milky way in half a year):

Timing is everything. lol

Well, since it's a fantasy, anyway....I'll travel backwards in time, so I can meet the Pharaoh Hatshepsut, Cleopatra, Richard Feynmann, Abraham Lincoln, Elizabeth I and a few others. (Implicit in this fantasy is my ability to fluently speak ancient languages.) (I'd also like to meet my father, since I never really knew him. THAT would be FUN!) My fantasy feet will remain planted on terra firma, thank you!

Whoopsie. Explains why no one shows up to those time traveler conventions and why we have so much space debris?