There has been only one instance of the Governor-General not taking the Prime Minister 's advice . Governor-General Sir John Kerr , acting on his own , dismissed Prime Minister Gough Whitlam in the 1975 Australian constitutional crisis . There has been only one instance of the Governor-General acting outside the advice of the Prime Minister of the day , when Governor-General Sir John Kerr , acting on his own authority , dismissed Prime Minister Gough Whitlam in the 1975 Australian constitutional crisis .