The Canadian-born German former professional ice hockey coach was set to be out of contract at the end of June.
Press Association Sport understands Krueger, 59, has left his position with immediate effect.
Southampton added “more information about the future leadership structure” of the club would be released at the end of the season.
The statement continued: “For now, all our efforts and focus are on a strong finish to the season and remaining within the Premier League.
Krueger joined Southampton in 2014 and helped stabilise the club in the Premier League.
“Everything has a beginning and everything has an end. It is the journey in-between that I will treasure. Thank you Southampton Football Club for the journey,” Krueger said.
Majority shareholder Gao Jisheng praised Krueger’s work. He said: “Ralph represented our club with honour and integrity, and followed ‘The Southampton Way’ with great passion.
Fellow shareholder Katharina Liebherr added: “Ralph laid the foundations for the Saints to be a sustainable team that put pressure on the top sides in the Premier League.