 He was the youngest son of eleven children, seven daughters and four sons, born to George, 2nd Duke of Sutherland (and 20th Earl) by his wife Lady Harriet Howard.His surviving siblings included Lady Elizabeth Georgiana (1824–1878), who married the 8th Duke of Argyll, Lady Evelyn Gower (1825–1869), who married the 12th Lord Blantyre, Lady Caroline Gower (1827–1887), who married the 4th Duke of Leinster, George Gower (1828–1892), who became the 3rd Duke of Sutherland, Lady Constance Gower (1834–1880), who married the 1st Duke of Westminster, Lord Albert Gower (1843–1874), married Grace Abdy.His paternal grandparents were George Leveson-Gower, 1st Duke of Sutherland and his wife Elizabeth Gordon, "de jure" Countess of Sutherland.