When teenage Voldemort came to his mother's house, his uncle Morfin mistook him for his father Tom Riddle (at first sight) however Morfin and Tom rarely used to see each other.

> 
> Morfin pushed the hair out of his dirty face, the better to see
> Riddle, and Harry saw that he wore Marvolo's black-stoned ring on his
> right hand.
> 
> 
> "I thought you was that Muggle," whispered Morfin. "You look mighty
> like that Muggle."
> 
> 
> "What Muggle?" said Riddle sharply.
> 
> 
> "That Muggle what my sister took a fancy to, that Muggle what lives in
> the big house over the way,"
> 
> 
> 

There's a line in Half Blood Prince stating

> 
> Merope had got her dying wish: he was his handsome father in miniature
> 
> 
> 

But Riddle's own gardener Frank couldn't make any connection between the two faces and told the police that a stranger was the only person he saw around.

> 
> Frank was stubbornly repeating, again and again, that he was innocent,
> and that the only person he had seen near the house on the day of the
> Riddles' deaths had been a teenage boy, a stranger, dark-haired and
> pale.
> 
> 
> 

Frank was working for Riddle's ever since he returned from war. Why didn't he recognize his master's face double?