Unsupervised learning to optimize a function of the input I am looking to build a neural network that takes an input vector $\mathbf{X}$ and outputs a vector $\mathbf{Y}$ such at $f(\mathbf{X}, \mathbf{Y})$ is minimized, where $f$ is some function. The network will see many different $\mathbf{X}$ during training to adjust its weights and biases; then I will test the network by using the test set $\{x_1, \dots, x_n \}$ to calculate $\sum(f(x_1, y), \dots, f(x_n, y))$ to see if this sum is minimized.

However, I have no labels for the output $\mathbf{Y}$. The loss function I am trying to minimize is based on the input and output instead of the output and label. I tried many standard Keras and TensorFlow loss functions, but they are unable to do the job. Any thoughts on how this might be achieved?
According to your description, you already know your function $f$ to be optimized. So you should use it directly instead of the standard loss functions. In this other post there is an explanation of how to use $f$ as a custom loss function in Keras.