Groom of the Bedchamber. Despite his earlier break with Newcastle, FitzRoy continued to work with Grafton. He spoke in Parliament against the 1765 Regency Bill in April of the year, and in May worked as an intermediary between Grafton and fellow Whig William Pitt in the organisation of Lord Rockingham's term as prime minister. With Rockingham coming into power, Grafton was made Secretary of State for the Northern Department and FitzRoy found new favour in his military career. He was made colonel of the 14th Dragoons on 11 September of the same year because of this, and began serving as Vice-Chamberlain to Queen Charlotte in 1768. Grafton then became prime minister, a position he would hold until 1770. Towards the end of 1769 FitzRoy fell out with Grafton again, as a result of which he refused an appointment as Vice-Chamberlain of the Household in 1770. Baron Southampton FitzRoy was promoted to major-general in 1772, and on 20 October transferred his colonelcy from the 14th Dragoons to that of the 3rd Dragoons. He moved as an MP from Bury St Edmunds to Thetford in 1774, at which point he began to operate more independently in politics. When Grafton moved to the opposition over the prime minister Lord North's aggressive policies towards British North America in October 1775, FitzRoy again refused to join him. He was promoted to lieutenant-general on 29 August 1777. As a reward for his support of North's policies FitzRoy was created Baron Southampton on 17 October 1780. In December 1780 FitzRoy was appointed Groom of the Stole to the Prince of Wales. In this role he was a special advisor and head of household to Wales. FitzRoy acted as an intermediary between the prince and his father George III during the disagreements surrounding the prince's secret marriage to Maria Fitzherbert. He was also heavily involved in the troubles, and attempted solutions, surrounding the prince's very large debts, much of which came from the construction of Carlton House. One contemporary said of FitzRoy that he "had only one fault, to be dislik'd by [the Prince of Wales] and a blind attachment to the king". Now serving in the House of Lords, FitzRoy created the motion for the loyal address to the king at the opening of parliament in 1781, and in 1782 returned to his military roots when Sackville, now known as Germain, was created Viscount Sackville. This was