Video for Windows Codec x264 and VP80
There isn't really an alternative to x264vfw. Are you sure the error is with the codec and not your application? It's in wide use, such a major bug would definitely be found.

Also be aware that h264 doesn't work in a frame by frame basis unless you encode only intra frames.
VP80 is NOT h264, just a similar codec.

If you need to develop an app you probably want to turn to a commercial h264 implementation like MainConcepts Codec SDK: http://www.mainconcept.com/products/sdks/codec-sdk/video/h264avc.html
Should also come with a vfw codec.

Alternatively you can turn to GStreamer which offers C# bindings: http://cgit.freedesktop.org/gstreamer/gstreamer-sharp

Another option could may be ffdshow it adds quite a few codecs to VfW and I believe also x264.