Deputy Hunter Adam handed the Bailiff Richard Collas a letter informing him of the decision following a recess for his board to discuss its options.
Deputy minister Barry Brehaut and deputies Elis Bebb, David Inglis and Arrun Wilkie have also resigned.
Earlier, deputies voted to allow the department to overspend by Â£2.5m.
However, that unanimous decision came despite severe criticism of the department's handling of its finances and its decision in November to close wards and postpone non-emergency operations.
Deputy Mark Dorey, a former Social Security minister and a current member of Treasury and Resources, was voted to replace Deputy Adam.
He beat the current Housing Minister Dave Jones by 27 votes to 16.
During his speech to members, Deputy Dorey said: "I believe that I would bring a fresh approach to the role, characterised by energy, openness and willingness to listen."
States members adjourned, following his election to allow time for deputies to discuss nominations for the remaining four seats.
They went unopposed to Deputy Sandra James and Deputy Martin Storey and two former board members, Deputy Elis Bebb and Deputy Barry Brehaut.
All were proposed by the new minister and their nominations were seconded by the unsuccessful ministerial candidate Deputy Jones.
Guernsey's minister for Health and Social Services has resigned, ahead of a planned States debate on a motion of no confidence.