The East Superleague side fell behind when Paul Quinn fired in from a corner for his second goal in two games.
Rose drew level through a Jamie Reckord own goal, but Graham and Alex Schalk put the Premiership side in control.
However, the semi-professionals refused to buckle and Ruari MacLennan scored his fourth cup goal before Graham pounced to finally settle the tie.
While County move into the quarter-final draw, the first junior side to reach the last 16 of the competition were far from disgraced.
It is an indication of the strength of his squad that County manager Jim McIntyre was able to make six changes to the side that beat Celtic in last week's League Cup semi final.
There was certainly no sign of disruption early on as the Premiership outfit raced into the lead in their first serious attack after a bright start by the visitors.
Colin Leiper nearly headed Ian McShane's corner into his own net with Ryan Baptie clearing off the line, but the ball broke to Quinn, who hammered home from 12 yards for his second goal since his return from Aberdeen.
Many would have expected the Rose to wilt after that fourth-minute setback, but they showed real character to take the game to the Staggies and create a couple of decent chances.
Conor Kelly fired a shot over the bar from the edge of the box then Tommy Coyne swapped passes with Graham Weir before forcing Gary Woods into a save from a similar distance.
County's main threat continued to be from set pieces, with Quinn and Raffa De Vita going close, while Linlithgow goalkeeper Ally Adams made an outstanding diving save low to his left to prevent Brian Graham converting Jonathan Franks' inviting cross.
Rose have shown resilience throughout their Scottish Cup adventure and, after Ruari McLennan then Blair Batchelor had shots blocked, Scott Boyd blasted it off the goal-line against Reckord and into the empty net to spark wild celebrations among the Rose players and their large travelling support.
They continued to give as good as they got at the start of the second half, with Batchelor forcing Woods into a save, but the Premiership side's superiority at set pieces paid off.
McShane whipped in another corner from the left and Graham raced clear to head home from six yards.
Schalk finally killed off the tie five minutes later as Jackson Irvine found space in the box to head on for the Dutchman to slam the ball in from close range.
Back came the junior side with a slick passing move climaxed by MacLennan's finish in 75 minutes only for the Staggies to spoil any chance of a fairytale finish.
Graham scored his seventh goal of the season with a shot from the edge of the area to leave Rose to concentrate on next week's Scottish Junior Cup tie away to Kilwinning Rangers.
Brian Graham's double helped Ross County end junior side Linlithgow Rose's brave Scottish Cup run.