On a green surface and in damp, overcast conditions, Australia fell to 8-4 and 17-5 after losing the toss.
Vernon Philander took 5-21, Kyle Abbott 3-41 and though Steve Smith ended 48 not out, Australia slumped to their lowest home total against the Proteas.
South Africa reached 171-5 by the close for a lead of 86 - Hashim Amla hitting 47, while Mitchell Starc has 3-49.
The tourists, without injured skipper AB de Villiers and pace spearhead Dale Steyn, already lead the three-Test series 1-0.
This Test is the first time since both made their debuts against England in December 2004 that South Africa have played without either of them.
In the absence of Steyn, Philander tore through the Australia top order, starting with David Warner edging a drive to a delivery so wide he could barely reach it.
Joe Burns was lbw to Abbott and, after Usman Khawaja and Adam Voges edged behind the wicket, debutant Callum Ferguson was run out by a direct hit from substitute fielder Dane Vilas for three.
From five wickets down, Australia needed captain Smith to get past their lowest home total of 58, but they were still shot out for the worst score in a Hobart Test.
Subscribe to the BBC Sport newsletter to get our pick of news, features and video sent to your inbox.
Australia were bowled out for only 85 on the first day of the second Test against South Africa at Hobart.