Why can Voldemort touch Harry?

I've read all the books, but it still confuses me how Voldemort can touch Harry in the graveyard in the fourth book because he used his blood, but then he can't touch him until he leaves the house in the last book?