The goal is to see if a computer can beat four of the best players in the variant of poker known as no-limit heads-up Texas hold 'em.
In a similar tournament in 2015, the humans won.
The algorithm could be adapted for use in medicine, cybersecurity, business and the military, its creators said.
The matches - dubbed Brains v Artificial Intelligence -  are being held over 20 days at Rivers Casino in Pittsburgh.
The four human players are matched against Libratus, an artificial intelligence program developed at Carnegie Mellon University's (CMU) school of computer science.
It uses algorithms built over 15 million core hours of computation at the Pittsburgh Supercomputing Centre to analyse the rules of poker.
Viewers can watch the entire event live on video-streaming games site Twitch, where there will be a stream for each of the human players:
The players are vying for shares of a $200,000 (Â£162,000) prize, while the CMU scientists are hoping to set a new benchmark for artificial intelligence.
"Since the earliest days of AI research, beating top human players has been a powerful measure of progress in the field," said Tuomas Sandholm, professor of computer science at Carnegie Mellon.
"That was achieved with chess in 1997, with Jeopardy in 2009 and with the board game Go just last year.
"Poker poses a far more difficult challenge than these games, as it requires a machine to make extremely complicated decisions based on incomplete information while contending with bluffs, slow play and other ploys."
A similar contest in 2015, with a computer program dubbed Claudico, failed to beat the humans, with scientists concluding that the 80,000 hands it had played had proved too few to establish its superiority.
This time around, there will be 120,000 hands.
Claudico also made some obvious bluffs that they had been able to exploit, the players said.
This time around, the AI will be able to bluff with precisely calculated values to balance risk and reward.
It will also be able to deploy random moves in a way that human players would have great difficulty doing.
The scientists hope that it could baffle human opponents, with unusual strategies such as making tiny bets or massive over-bets.
Player Jason Les said of the new contest: "I'm very excited to see what this latest AI is like."
"I thought Claudico was tough to play; knowing the resources and the ideas that Dr Sandholm and his team have had available in the 20 months since the first contest, I assume this AI will be even more challenging."
Solving the complex game of poker had many real-world applications, Dr Sandholm said.
"Extending AI to real-world decision-making, where details are unknown and adversaries are actively revising their strategies, is fundamentally harder than games with perfect information or question-answering systems," said Nick Nystrom, senior director of research at Pittsburgh Supercomputing Center .
"This is where it really gets interesting."
An unusual poker game is taking place in Pittsburgh, pitting human players against  an artificial intelligence program.