How to encode with x265 cli
Your source is neither YUV nor Y4M, going by the extension (and also the FFmpeg error you received). x265 is referring to a raw uncompressed bitstream by those extensions, not simply the pixel format. A .h264 is a raw H.264-encoded bitstream.

To use ffmpeg only, start the cmd like this:

ffmpeg -f h264 -i video.h264 -c:v libx265 ....


To use the x265 standalone encoder, use

ffmpeg -f h264 -i video.h264 -f yuv4mpegpipe - | x265 --input - --fps 24 ....