In Out of Sheer Rage: Wrestling with D. H. Lawrence (1997), Geoff Dyer answers:
I also thought of knocking on the door of our old house, explaining that I was born there, that I lived there until I was eleven, and wanted to look around. I abandoned the idea as soon as I’d thought of it. Houses have no loyalty. We can live in a place ten years and within a fortnight of moving out it is as if we have never been there. It may still bear the scars of our occupancy, of our botched attempts at DIY, but it vacates itself of our memory as soon as the new people move their stuff in. We want houses to reciprocate our feelings of loss but, like the rectangle of unfaded paint where a favourite mirror once hung, they give us nothing to reflect upon. Often in films someone goes to a house where he once spent happier times and, slowly, the screen is filled with laughing. This convention works so powerfully precisely because, in life, it is not like that. It testifies to the strength of our longing: we want houses to be haunted. They never are.