It had been visited by Henry in 1535 , with his second wife Anne Boleyn , but had been empty and unattended for some time . In 1535 , Henry VIII visited the castle with his second wife Anne Boleyn , which had been empty and unattended for some time .