What type of neural network could emulate a binary to HTML conversion tool? I've got a problem, which I thought could be solved by using a neural network:

I've got a binary file and a tool that converts that file into a readable html file (probably a text file as well). How the tool is converting that file is a secret and it is too complex to reverse engineer (>100000 lines of assembler for this function) and the binary files are too big and complex to interpret for a human in a feasible time. My idea was to teach a neural network with those binary files and the known output to train the network, how to convert these files. The binary files differ extremely in size(from a few kb up to serval mb).

I've got some basic knowledge of neural networks but I'm not sure where to start to search for the right approach. A friend of mine (with a bit more basic knowledge) said this is definitely doable and suggested a seq2seq model in tensorflow, but after some research I'm not sure if this is the right one for my problem.
In short your friend is correct, seq2seq is a reasonable match to the problem. 

However, from your numbers this could be too complex to use current machine learning libraries on. 

Despite you calling it "not feasible", you are far better off with the reverse engineering effort in my opinion.

If you really want to have a try using ML, you could start with some shorter files with the generated output and attempt to train a LSTM-based seq2seq model to match them. It may help, if the HTML is strongly templated and driven by data in the binary file, to reverse-engineer just the templating, and use seq2seq model to derive a more raw data sequence as opposed to HTML. 

You should expect to spend several weeks learning how seq2seq works, followed by an unknown amount of time in the attempt to apply it to your problem. You will need a large number of sample files to train your model - hundreds of thousands, maybe millions. There might be exceptions if the generated HTML is very repetitive/formulaic, or if short segments of binary data can also generate short segments of HTML (so you could train on smaller segments of the larger files).



If any of this would break the terms and conditions for use of the software, I advise you not to attempt either reverse engineering or machine learning.