So far I've been walking everywhere I go. I did use a carriage once to get to Winterhold. I was trying to get there on foot and was never able to find a way up, so I fast traveled back to a city and hopped a carriage.
I'm actually enjoying walking though. It is time consuming, but along the way you find unexplored caves and forts. So, it's worth my time to walk. I haven't got a horse yet; not bought or stolen one. I might get one later, after I buy my house.
But, I'm going to go with "all of the above" with a strong preference towards walking.