 When Charlotte was eight , her father , whose affections had returned to Fitzherbert , decided that he wanted Carlton House to himself . He took over his wife 's apartments ( Caroline received space in Kensington Palace instead ) , and moved their daughter into Montague House , adjacent to Carlton House . As James Chambers , another Charlotte biographer , put it , the young Princess " lived in a household of her own , in the company of no one who was not paid to be there " . The move took place without the presence of Charlotte 's governess , Lady Elgin ( widow of Charles Bruce , 5th Earl of Elgin ) , with whom she was very close . Lady Elgin had been forced to retire , ostensibly on account of age , but most likely because George was angry that Lady Elgin had taken Charlotte to see the King without George 's permission . George also dismissed the sub @-@ governess , Miss Hayman , for being too friendly with Caroline — and the Princess of Wales promptly hired her . Lady Elgin 's replacement , Lady de Clifford ( widow of Edward Southwell , 20th Baron de Clifford ) , was fond of Charlotte , and too good natured to discipline the child , who had grown into an exuberant tomboy . Lady de Clifford brought one of her grandsons , the Honourable George Keppel , three years younger than Charlotte , as a playmate for her . Forty years later , Keppel , by then Earl of Albemarle , would remember Charlotte in his memoirs , the source of many of the anecdotes of Charlotte as a small girl . In addition to tomboy tales of horses and fisticuffs , he remembered them seeing a crowd gathered outside the Keppel house at Earl 's Court , who were hoping to see the young Princess . The two children went outside and joined the crowd , unrecognised . 
