What kind of optimization problem this belongs to? Supervised learning is: given $X$ and $Y$, we want to find a good predictive function $f$ such that $f(X)$ is "close" to $Y$. Examples include XGBoost (ensemble of decision trees), neural networks, $\ell_1$-regularized linear fit, etc.

In the problem I am encountering I have $X$. Each row of $X$ describes each day's scenario, for example, the financial market descriptors of that day. I don't have $Y$. $Y$ is the decision for each day, for example, whether I should buy/hold SP500. For each $Y$, I will have some scoring, say score(Y) system to describe whether my decision is good over a long period of time. Now, I want to find simple, robust $f$ such that $f(X)$ gets the low score. Of course, for arbitrarily complex $f(X)$, the score score(f(X)) can be very low but the higher complexity generally means worse model for the problem I am tackling on. So there should be some regularization parameter that I could tune to control the complexity of $f$. The $f$ can be anything, like supervised learning. It could be ensemble of decision trees (XGBoost, random forest, etc...), linear (L1-regularized fitting etc.). I am currently essentially hard coding my problem into a optimization problem to solve on. But I think there should be a field out there tackling such problems.

If so, what is this type of problems generally called. And what's the state of the art packages currently in python. Thank you.
Yes, many standard methods can handle this.  Your score is called a loss function.  In particular, let $\ell(x_i,y_i)$ denote the penalty (score) if on input $x_i$, your decision rule outputs $y_i$.  If $x_1,\dots,x_n$ is a training set, we define the total loss over the training set (the empirical risk) of a decision rule $f$ to be

$$L(f) = \sum_{i=1}^n \ell(x_i,f(x_i)).$$

Now your goal is to find $f$ that minimizes the total loss $L(f)$.  Many machine learning algorithms can be used in this manner, with an arbitrary loss function.  Typically, we require that the loss function be differentiable, and then we use gradient descent to find $f$ that minimizes $L(f)$.  For instance, if you are using neural networks, then you use gradient descent to find weights for the neural network that minimize $L(f)$.  Often you add a regularization term to the loss function, too, to prevent overfitting.