avoptions: Return explicitly NAN or {0,0} if the option isn't found

This actually matches what av_get_double did earlier, the
0.0/0.0 division was intentional, for producing NAN.

Still keeping the check for the return value from
av_get_number, for clarity.

Signed-off-by: Martin Storsjö <martin@martin.st>
