Result complement plus 过
Normally, it is either:

你[看过]那部小说没有? = [Had] you [read] that novel?

or 

你[看完]那部小说没有? = [Have] you [finished] reading that novel?


看(verb) + 过(verb particle) marking 'experiential' aspect = "had read"
看(verb) + 完(result complement) marking 'complete or finish' aspect = "finish reading"



  in "你看过完那部小说没有?" (had you read finish that novel?), '看过'(had read) already indicated you finished reading the book in the past, '完' is redundant here.


~


  in "你看完过那部小说没有？" (had you ever finish reading the novel?) '看完'(finish reading) already indicated you had finish reading it,  '过' is also redundant 
  here. This implies  'you had tried to read that novel before, but had never been successful in finish it.'


This question only make sense if someone pick up a novel and only read half of it, then few weeks later pick it up again, read it from the start and stop at the middle again. Kind of weird, right? 

~

A more sensible examples of [V] +[result complement]+ [过 (verb particle)]:

看见过 (had seen)


看(verb- to see) 
见 (result complement- and successfully seen)
过 (verb particle) - marking 'experiential' aspect- had experienced


conclusion: 

[Verb + 过] cannot be followed by '完'(redundant)

[Verb + 完 + 过] indicate "had finished (verb) before"