What is a Black Box attack against Machine Learning algorithms?
This is known as a &quot;model extraction attack&quot;, and there's a line of research in the computer security literature on this problem.  For instance, you might look at the seminal paper by Papernot et al on the subject.  A little time on Google Scholar should rapidly turn up dozens of papers on the subject.