What type of media server software do I need to transcode in near real time?
FFMPEG is what you're looking for.  You can use it to transcode most anything, it can easily be integrated via STDIO, and there are a handful of library versions of it you can use as well.

I'd recommend hopping over to StackOverflow.com, where this topic has been heavily discussed.