avcodec/dnxhd_parser: Do not return invalid value from dnxhd_find_frame_end() on error

Fixes: Null pointer dereference

Fixes: CVE-2017-9608
Found-by: Yihan Lian
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 611b35627488a8d0763e75c25ee0875c5b7987dd)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>