What should I study to find optimal value of best feature combinations in machine learning?
Welcome to DS.SE!

As you already mentioned, machine learning is not about finding the best values of features regarding a function. Also, loss functions usually measure the errors of predictions and not the quality of the feature values (e.g. " the best product y").

What you are looking for is called optimization or search algorithms but not learning or prediction. If the search space (i.e. the combination of all possible feature values) is not that large, you can just perform an exhaustive search (trying all combination). If the search space is large, then smarter approaches might be useful. These techniques need a function (usually called objective function) to optimize (finding the combination of input values that results in a min/max outcome). 

In general, there are two approaches: mathematical modeling (i.e. linear programming) and metaheuristics (e.g., evolutionary algorithms and swarm intelligence). Mathematical models can be fast and guarantee to find the optimum when the objective function is well-defined and the underlying assumption are met (you may need to simplify your constraints sometimes). If these conditions are not met, then metaheuristics usually can find a set of approximate solutions.