Challenging an Ambassador Exchange
After claiming Ambassador, the player should pause for a few seconds and confirm that there are no challenges before drawing new cards. The reason is so that, if successfully challenged, the player does not gain any improper information about the cards remaining in the deck. (For example, seeing two Dukes in the deck can be a powerful suggestion that the players claiming it are lying!)

(According to the rules, you always check for no challenge before proceeding with a claimed action. But for the Assassin and Captain, you're waiting for the victim to take the hit (or claim a blocker) anyway, and for the Duke, it's easy enough to just return the coins. Only for the Ambassador is it important to deliberately pause. 

It is also worth mentioning that you seem to have a slight detail confused. A player using Ambassador always draws 2 and returns 2, even if they have only one influence remaining.