A source close to Mr Gupta said the discussions had not been substantive but he would seek further talks when he flies back to the UK early next week.
Government help on energy costs and plant modernisation are issues he wants to discuss, according to the source.
The government declined to comment.
It comes as the government said it was ready to offer financial support to "clinch a buyer" to save the steelworks, where more than 4,000 jobs are at risk.
Tata Steel - which announced last week that it was selling its loss-making UK businesses - has said it will close the plant unless it finds a buyer.
German steelmaker ThyssenKrupp has also been touted as another potential buyer, according to reports.
Tata directly employs 15,000 workers in the UK and supports thousands of others, across plants in Port Talbot, Rotherham, Corby and Shotton.
It said it was "committed to running a meaningful process to explore strategic alternatives" for its UK business.
What's going wrong with Britain's steel industry?
Tata Steel UK: What are the options?
Is China to blame for steel woes?
Mr Gupta's commodities firm Liberty House has already saved several UK steelworks.
A source close to the businessman said he was "definitely interested" in Tata's "downstream" operations and was looking at options which would keep liquid steel-making in the UK.
BBC political correspondent Carole Walker said Mr Gupta wanted to discuss possible government help to replace Port Talbot's traditional blast furnaces with modern electric arc furnaces and energy costs.
The source said it would not be an open-ended commitment, would protect most of the jobs at the site and could make it more profitable in the long term.
Extra relief from carbon tax is another important issue, according to the source.
On Sunday's Andrew Marr show, Business Secretary Sajid Javid said he would rule nothing out, but he maintained the government's view that nationalisation was not the answer to the crisis.
It wants to find a commercial buyer for the Port Talbot plant, saying this will offer a "long-term, viable future".
Mr Javid signalled that ministers were working on plans to reduce energy costs and take on some of the pension liabilities to make a purchase more attractive to investors.
Tata Steel has said there is "no fixed timeline" for the sale process but stressed that urgency is needed to avoid "a long period of uncertainty" employees and customers.
It said the sale process began on 30 March and added: "Tata Steel Europe is in the process of finalising the appointment of advisers and will soon launch a process globally of seeking an investor for the UK operations."
Labour says the government should be prepared to take the Port Talbot plant - which Tata says is losing Â£1m a day - into public ownership to safeguard its future until a buyer can be found.
Shadow chancellor John McDonnell suggested the government could maintain a stake in the plant after a sale.
He has also urged ministers to speed up infrastructure projects, such as HS2, to support the steel industry and encourage investment.
Mr Javid - who was criticised for being in Australia during Tata's crunch meeting in Mumbai last Tuesday - admitted that Tata's announcement had caught the government by surprise
He said the government had known "a few weeks ago" that Tata was reviewing its UK operations and persuaded it not to go ahead with the immediate closure of Port Talbot - but its decision to sell off its UK assets had gone "much further than we expected".
Mr McDonnell said Mr Javid should not continue in his role as business secretary.
He told the Radio 4's World This Weekend programme someone "more dynamic" and "effective" was needed.
Ministers have faced criticism for failing to take more action to prevent the "dumping" of cheap Chinese steel - selling it cheaply at a loss - seen as one of the key reasons for the problems in the UK steel industry.
On Marr, Mr Javid said the UK had been "the leader" in pushing for action on tariffs but he said protectionism "is in no-one's interests".
Elsewhere, the government has played down the impact of new Chinese import tariffs of up to 46.3%.
David Cameron and Welsh First Minister Carwyn Jones will meet on Tuesday to discuss the situation.
Tycoon Sanjeev Gupta has held initial talks with the government over the possible purchase of parts of Tata Steel's UK business, including Port Talbot steelworks, the BBC understands.