Usually, when people speak of time travel, they mean traveling at a speed or direction that significantly deviates from the norm. At this point, you and I are traveling at slightly different speeds in time relative to each other. I personally think that it's likely we will find a way to enact some form of time travel, as we have already discovered that we can affect, albeit minimally, the rate at which time proceeds, through speed and gravitational force.
And fourstaff, just because we have not recognized anyone traveling backwards in time to meet us does not mean that it has not happened. Anyone from a future in which humanity had access to time travel technology would also likely have the capability to effectively disguise themselves. One must also consider the fact that future people would not want to affect history, because it could affect their future. Both forwards and backwards travel in time do throw up logical problems, though. Going into the past, the most obvious one is the "I killed my grandmother" impossibility, in that if you killed your grandmother, you would cease to exist, and then you could not have killed your grandmother. With travel into the future, things stay mostly fine unless you travel back into the past. For example, if you went two hundred years into the future and brought back the theory for faster-than-light space travel, and then you were the pioneer of that theory in your time, the actual information in that theory would have come out of nowhere, which is impossible.
Who knows if we will ever be able to travel it the fourth dimension, but I think it certainly makes for many interesting possibilities.