The 32-year-old knows Coleman's side will be tough to beat having improved hugely since two defeats by Serbia in the last World Cup qualifying campaign.
The Chelsea defender, who has led Serbia to the top of Group D, says they face a difficult match ahead of them.
"This is the hardest game since we started qualification," he said.
Ivanovic scored in the 6-1 thrashing of Wales in 2014 World Cup qualifying at Novi Sad, in September 2012 - the heaviest defeat in Coleman's reign.
A year later he was part of the side which won 3-0 at the Cardiff City Stadium - the last time Wales lost a home qualifier.
Set to earn his 91st cap for his country on Saturday, Ivanovic believes the scars of the past will provide extra motivation for Wales.
"We know that and we have to be careful with that. We have to play much much better than last time to get something from the game because Wales is a completely different team from that moment," he continued.
"Those kind of games happen may be once in 100 years. It's very difficult to win against one team three times in a row even when the opponent is not that strong and you cannot say that about Wales."
Serbia go into the game topping the group as they edge the Republic of Ireland on goal difference while Wales are two points adrift in third.
The visitors will be without the suspended Manchester City defender Aleksander Kolarov, who is suspended after collecting two bookings in the group.
Wales will welcome the return of Arsenal midfielder Aaron Ramsey for the first time in this campaign after his recovery from a hamstring injury.
Ivanovic believes they will be stronger for his inclusion, but inevitably the major threat will be Real Madrid star Gareth Bale.
"I think he is the biggest name and biggest star in the Wales team. But I think Wales as a group have improved," Ivanovic said.
"The team have played a couple of years together to bring results and will, of course, be very difficult to stop Gareth Bale and the rest of the guys."
"Most important for us is we have a lot of new things - new coach, new shape of the team. We are still building. We have improved a lot from last summer and we are going in the right direction.
"But the way to Russia is so long we have to be careful and we have to play every single game like a final.  We know this is our hardest game so far."
Branislav Ivanovic says Wales will want to avenge their previous two defeats against Serbia when they meet in Saturday's World Cup qualifier.