Recommandation for on-premises video file transcoding server accessible via REST API?
Telestream used to make one called FlipFactory that I have used (a long time ago). I believe it's called Vantage now. https://www.telestream.net/vantage/overview.htm
A lot of people do this kind of thing in the cloud now using things like Amazon's Elastic Transcoder 
https://aws.amazon.com/elastictranscoder/