vf_pullup: simplify, fix double free error

The memory allocation for f->diffs was freed multiple times in some
corner cases. Simplify the code so that this doesn't happen.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
