In 1817, Olive wrote a letter to the Prince Regent, claiming that she was the natural daughter of Prince Henry, Duke of Cumberland by Mrs. Olive Payne (who was James Wilmot's sister and her actual aunt).
She herself, ten days after her birth, was, she alleged, taken from her mother, and substituted for the still-born child of Robert Wilmot.