Later the castle was the occasional residence of the governor of the Isle of Wight and it became home to Princess Beatrice , youngest daughter of Queen Victoria , when she was governor . Afterwards his two youngest children were confined in the castle , and the Princess Elizabeth died there . Most recently it was the home of The Princess Beatrice , daughter of Queen Victoria , as Governor of the Isle of Wight , 1896-1944 .