What is the best machine learning algorithm to select best 3 variable combinations?
What you have is an instance of a local search or black-box optimization problem.
There are many AI algorithms to do this, though as commenters have noted, with only 10 binary variables, you're happier to just look at all $2^{10}$ of them.
If you had, say, 60, then looking at them all is no longer a good idea ($2^{60}$ is very large).
The simplest BBO technique is called hill climbing. It looks like this:

Pick a random subset of the variables to be true.
For each variable, determine the effect of flipping that variable from true to false in the current assignment.
Select the 'flip' that leads to the largest value of your function.
If the 'flip' has a larger value of your function than the current assignment, GOTO 2.
Otherwise, return the current assignment.

This amounts to: 'Pick an assignment at random, then keep making small changes that improve it until you can't anymore. Claim that this is the best solution.'
Clearly, this algorithm will not recover the globally best solution with certainty, but you could run it many times, and eventually, it will find it with high probability.
A complete algorithm, that will always uncover the global optimum, is simulated annealing. This algorithm has a temperature parameter, usually denoted T, which changes according to a temperature schedule. If you start T large 'enough', and decrease it slowly 'enough', then it will find the global optimum. Unfortunately, 'enough' depends a lot on the shape of the function you are optimizing, which you generally cannot observe. There are some common guesses for T, like multiplying it by, e.g. 0.9999 at each step in the search, that seem to work okay in practice.
Both of these algorithms are widely available in packages.