Will there be a noticeable offset if knitting rib from an unravelled edge?
You're correct in that the fluffiness of the yarn and the colour of the jumper should hide the offset to some degree. Most people won't pick up on the change unless they are looking really closely (and really -- who does that?). 

TECHknitting did a series on grafting that addressed ways to disguise this particular dilemma. If you haven't already read it, you may find it helpful. Between these tricks and the yarn itself, no one will ever know :)