Governor George Gipps was disturbed when he heard about it, but La Trobe convinced him that he had acted innocently. On that land, La Trobe erected his home, which he had transported from London in sections, and which is preserved as LaTrobe's Cottage. Melbourne had a population of around 3,000 at the time and was rapidly expanding. La Trobe commenced works to improve sanitation and streets. As the Port Phillip District was a dependency of New South Wales, all land sales, building plans and officer appointments had to be approved by Gipps, with whom La Trobe had a good personal and working relationship. A Separation Association had been formed in 1840, with the aim of making the Port Phillip District a separate colony. In 1841, La Trobe wrote to Gipps, asking him to visit Melbourne to form his own opinion on the separation question. La Trobe did not actively campaign for separation, being content that Earl Grey had included separation in the reorganisation plan for the colonies. La Trobe also acted as lieutenant-governor of Van Diemen's Land for four months in 1846–47. In July 1851, the Port Phillip achieved separation from New South Wales, becoming the colony of Victoria, and La Trobe became lieutenant-governor – a position he held until 1854. Soon after separation, gold was discovered at several locations in Victoria. La Trobe suddenly had to deal with the mass exodus of the population of Melbourne to the gold fields, as well as the later arrival of thousands of immigrants from other Australian colonies and overseas. He was commonly referred to as "Charley Joe", and by extension, any government officials or policemen were called "joes". Having tried, with varying degrees of success, to cope with the enormous population and economic expansion of the new colony, La Trobe, who suffered self-doubt and criticism due to his inexperience, submitted his resignation in December 1852, but had to wait until a replacement, Charles Hotham, could take his place. After Victoria Towards the end of his governorship, La Trobe's wife, Sophie, became ill and returned to Europe with their four children. She died on 30 January 1854. On his return to Europe after his term, La Trobe married Sophie's sister, Rose Isabelle de Meuron (1821–1883) in 1855, a marriage which was illegal in English law, being considered incestuous at the time. (See Deceased Wife's Sister's Marriage Act 1907.) The couple had two daughters