Having failed to score in their last six home matches, Saints took the lead after only 10 minutes when Manolo Gabbiadini side-footed past Joe Hart.
West Ham's problems then spiralled, as Marko Arnautovic was sent off for elbowing Jack Stephens, before Dusan Tadic made it 2-0 from the penalty spot after Jose Fonte brought down Steven Davis.
Summer signing Javier Hernandez thought he had snatched an unlikely comeback for the visitors with two goals from close range either side of half time.
But the Hammers conceded a second penalty when Pablo Zabaleta bundled into Maya Yoshida and substitute Austin coolly slotted past Hart.
Referee Lee Mason had some important decisions to make throughout the dramatic match at St Mary's - and he got most of them right.
Mason was left with little choice but show a straight red to Arnautovic on 33 minutes, after the Hammers' record signing needlessly raised his elbow into Stephens' neck. The Austria international later apologised to the fans for his dismissal.
The first penalty was also an easy one to give, as former Saints defender Fonte pulled down Davis, but the second spot-kick in the 91st minute was a more contentious matter.
Zabaleta caught a rising Yoshida in the back, as he went to head the ball from close range, and Mason decided the Hammers defender had impeded his Southampton counter-part.
West Ham had earlier had a handball appeal against Stephens waved away, while the referee did not punish Mark Noble for his rash challenge on Saints midfielder Mario Lemina, who was making his debut following his £18.1m move from Juventus.
Before facing the Hammers, Southampton had not scored on home soil since 5 April - 576 minutes of football.
They had 29 shots against Swansea last week, landing only two on target in their goalless draw, so there was a collective sigh of relief around St Mary's when Gabbidani slotted home after Nathan Redmond's through ball.
Saints were only able to score from open play once during the match, but Gabbiadini and Redmond both went close with long-range efforts.
Substitute Shane Long had the ball in the net in the second-half but referee Mason had already blown for a foul on goalkeeper Hart in the build-up.
Media playback is not supported on this device
With their home ground out of use until next month following the athletics World Championships, it was always going to be a testing start to the season for West Ham.
They started their run of four consecutive away matches with an embarrassing 4-0 defeat against Manchester United last weekend, but put in a much-improved display at St Mary's - even after going down to 10 men.
Hernandez, who arrived for £16m from Bayer Leverkusan, lived up to his reputation as a potent poacher with his first two goals for his new club.
The Mexico international tapped in West Ham's opener after Fraser Forster parried Michail Antonio's shot into his path, and did the same again for his second goal - this time after Diafra Sakho's header came off the woodwork.
Boss Slaven Bilic said of his new striker: "We knew what we were getting with Hernandez: a really great player, great character, team player, really enthusiastic.
"You're expecting goals from him but on top of the goals, his determination; he sacrificed for the team in the best possible way."
While Hernandez looks to have settled in well, on-loan goalkeeper Hart still looked nervous at times and his confidence may be dented further after conceding seven goals in two matches.
Media playback is not supported on this device
Marko Arnautovic posted a message on social media after the game: "I want to apologise to all of the West Ham fans. Just want to let you know that I didn't want to let me team down and you fans. A big apology to you all."
West Ham manager Slaven Bilic on Arnautovic's sending off: "It is a big blow because you have to play with 10 men for a big part of the game. We have spoken about everything and that will stay inside the dressing room."
Southampton manager Mauricio Pellegrino: "It was a difficult game because we had everything under control after a sending off and two goals. This is football.
"West Ham are a strong team and they won a lot of balls in our box. It was a nice game to see, but difficult for a manager to be in."
Southampton host Wolves in the EFL Cup on Wednesday (19:45 BST), before returning to league action against Huddersfield on Saturday, 26 August (15:00 BST).
West Ham are also in EFL Cup action on Wednesday at League Two side Cheltenham Town (19:45 BST) before travelling to Newcastle on Saturday (15:00 BST).
Match ends, Southampton 3, West Ham United 2.
Second Half ends, Southampton 3, West Ham United 2.
Goal!  Southampton 3, West Ham United 2. Charlie Austin (Southampton) converts the penalty with a right footed shot to the centre of the goal.
Pablo Zabaleta (West Ham United) is shown the yellow card for a bad foul.
.
Penalty Southampton. Maya Yoshida draws a foul in the penalty area.
Penalty conceded by Pablo Zabaleta (West Ham United) after a foul in the penalty area.
Corner,  Southampton. Conceded by José Fonte.
Attempt blocked. Nathan Redmond (Southampton) left footed shot from outside the box is blocked. Assisted by James Ward-Prowse.
Foul by Ryan Bertrand (Southampton).
Joe Hart (West Ham United) wins a free kick in the defensive half.
Foul by Dusan Tadic (Southampton).
Mark Noble (West Ham United) wins a free kick in the attacking half.
Attempt blocked. Mark Noble (West Ham United) right footed shot from outside the box is blocked.
Foul by Nathan Redmond (Southampton).
Pablo Zabaleta (West Ham United) wins a free kick on the right wing.
Corner,  Southampton. Conceded by Pedro Obiang.
Attempt blocked. Oriol Romeu (Southampton) right footed shot from outside the box is blocked. Assisted by James Ward-Prowse.
Substitution, Southampton. Shane Long replaces Steven Davis.
Substitution, Southampton. Charlie Austin replaces Manolo Gabbiadini.
Attempt missed. Nathan Redmond (Southampton) right footed shot from outside the box is close, but misses to the left. Assisted by Steven Davis.
Foul by Maya Yoshida (Southampton).
Diafra Sakho (West Ham United) wins a free kick in the defensive half.
Nathan Redmond (Southampton) wins a free kick on the left wing.
Foul by Pablo Zabaleta (West Ham United).
Attempt missed. José Fonte (West Ham United) header from the centre of the box misses to the right. Assisted by Mark Noble with a cross following a set piece situation.
Foul by Nathan Redmond (Southampton).
Pablo Zabaleta (West Ham United) wins a free kick on the right wing.
Substitution, West Ham United. Pedro Obiang replaces Declan Rice.
Foul by Manolo Gabbiadini (Southampton).
Chicharito (West Ham United) wins a free kick in the defensive half.
Goal!  Southampton 2, West Ham United 2. Chicharito (West Ham United) right footed shot from the left side of the six yard box to the top left corner.
Attempt saved. Diafra Sakho (West Ham United) header from the centre of the box is saved in the top centre of the goal. Assisted by Aaron Cresswell with a cross.
Attempt missed. Diafra Sakho (West Ham United) header from the centre of the box misses to the right. Assisted by Chicharito with a cross.
Foul by Steven Davis (Southampton).
Diafra Sakho (West Ham United) wins a free kick in the defensive half.
Corner,  Southampton. Conceded by Aaron Cresswell.
Attempt missed. Oriol Romeu (Southampton) left footed shot from outside the box misses to the right. Assisted by Nathan Redmond.
Substitution, West Ham United. Edimilson Fernandes replaces André Ayew.
Substitution, West Ham United. Diafra Sakho replaces Michail Antonio.
Ten-man West Ham twice came from behind but Charlie Austin's injury time penalty gave Southampton their first victory under Mauricio Pellegrino.