configure: fix check for x264

libx264 recently started mangling the name of x264_encoder_open() to
prevent version mismatches, breaking our test.  Checking for another
function instead makes it work again.

Originally committed as revision 19754 to svn://svn.ffmpeg.org/ffmpeg/trunk
