How do I change output file format in ffmpeg batch, when using drag and drop?
With Gyan's help I was able to get it to work.
%~n1.mp4 will get the filename and substitute the file extension with mp4.
To get the correct path as well, I added %~p1.
So all in all I now use %~p1%~n1.mp4 and it works exactly as intended.