that would be the earl of orford , grey supposed ; walpole was the youngest son of the late prime minister , and had three brothers , but only the eldest had any influence-though a great deal less than his father had had .