Is it possible to use AI to reverse engineer software? I was thinking of something of the sort:


Build a program (call this one fake user) that generates lots and lots and lots of data based on the usage of another program (call this one target) using stimuli and response. For example, if the target is a minesweeper, the fake user would play the game a carl sagan number of times, as well as try to click all buttons on all sorts of different situations, etc...
run a machine learning program (call this one the copier) designed to evolve a code that works as similar as possible to the target. 
kablam, you have a "sufficiently nice" open source copy of the target.


Is this possible?

Is something else possible to achieve the same result, namely, to obtain a "sufficiently nice" open source copy of the original target program?
Remarkably, more or less the scenario you describe is not only feasible and has already been demonstrated (detailed explanation and fascinating videos at link).

However, the fidelity of the copy is currently quite limited:


So for now, your copy will be quite low quality.  However, there is a big exception to this rule: if the software you are copying is itself based on machine learning, then you can probably make a high-quality copy quite cheaply and easy, as I and my co-authors explain in this short article.

Interesting question and I'm quite sure that the correct answer will change rapidly over the next few years.