Morton are well placed to secure a Championship play-off spot after Tuesday's 1-0 win at Raith Rovers.
They were one of only two sides to earn a point at Ibrox in the league last season, and Duffy insists they will make it "as uncomfortable as possible".
"[A Morton win] would be a monumental upset," he told BBC Scotland.
"Rangers are a different proposition at Ibrox. Even though they drew last weekend [1-1 v Ross County], their home form has been very good over the last couple of years under Mark Warburton.
"The pitch, the expanse at Ibrox certainly suits the style they play. But we will certainly go there with a bit more heart after last night [the win at Raith].
"The heavy pitch took its toll so we have to make sure we recover right and prepare tactically as well.
"We will go there and try to make it as uncomfortable a day as we possibly can for Rangers. If we play well, manage to stifle Rangers a little bit and keep the ball, we can frustrate them."
With last season's Championship winners Rangers 27 points behind Celtic in the Premiership, Duffy believes Warburton's side will place extra emphasis on Sunday's fifth-round tie.
"The nature of being a big club like Rangers is the expectations are huge," noted the former Dundee centre-back and manager.
"They know it is their only chance of silverware this season. Celtic have already got the League Cup, and have all but wrapped up the league, so the only trophy available to Rangers and their supporters is the Scottish Cup.
"That will be the main objective and target now which will make it doubly difficult for us because all their focus will be on that.
"But we have got a job to do - to stay in the game and be difficult to beat. If Rangers get a little bit nervous, and their fans get frustrated, then you never know. But we have to be good enough to take advantage of those situations."
Media playback is not supported on this device
Morton's victory at Stark's Park lifted them to within two points of second-placed Dundee United and nine off Championship leaders Hibernian, but crucially, nine points clear of fifth-placed Queen of the South in the battle for a play-off place.
Duffy believes a prospect of a tilt at the title is unrealistic, insisting a play-off spot will be a significant achievement for the Greenock outfit.
"If we get into the play-offs, we will be singing and dancing, there is no doubt about that," he added.
"But it is still a long way away. A lot of clubs will be challenging for that. There will be clubs that can find a run of form and you can have a dip.
"But the players showed terrific character to bounce back from a disappointing weekend [losing to Queen of the South] and it sets us up nicely for the Scottish Cup tie against Rangers."
Greenock Morton boss Jim Duffy says it would be a huge shock if his side were to knock Rangers out of the Scottish Cup at Ibrox on Sunday.