How to optimize a simulation metric with deep learning without target values?
I am trying to use an RNN model that outputs bus routes and its input is the demand matrix. The bus routes are then used in a simulation which spits out a metric of how the routes performed. The question is, since there is no target value of bus routes, how do I back propagate the simulation result?

To explain the question with simple python code:

"""
The model is an RNN that takes 400,24,24 matrix as input

dimension 0 represents time, dimension 1 represents departure bus stop and dimension 2 represents the arrival bus stop. Each value is a count of the number of passengers who departed at a bus stop with an arrival bus stop in mind in a specific time

output is 64,24 matrix which will be reshaped to 8,8,24
dimension 0 is the sequence index, dimension 1 is the index of bus (there are 8 buses), dimension 2 is the softmaxed classifier dimension of 24 different bus stops. From the output, 8 bus stops are picked per bus with a sequence

These sequences are then used for path generations of buses and they are evaluated from a simulation
"""

model.train()
optimizer.zero_grad()
out = model(demand)#out is 64,24 demand is 400,24,24
demand, performance = simulation(out)#assume performance as float
#here the out has grad_fn but the performance does not
loss = SOME_NUMBER - performance
loss = torch.FloatTensor(loss)
#here I need to back propagate and it is the confusing part

#simply doing loss.backward() does nothing because no grad_fn

#out.backward() requires 64,24 gradients computed somehow from 1 #metric, causes complete divergence within few steps

optimizer.step()


How does the model output represent the bus routes? Maybe you could try a reinforced learning approach. Take a look at Deep-Q Learning, It basically takes and input vector (the state of the system) and outputs an action (usually represented by an index in your output layer), then it computes the reward of that action and uses it to train the model (without the need of target values).

Here are some resources that might help you get started:

https://towardsdatascience.com/double-deep-q-networks-905dd8325412

https://arxiv.org/pdf/1802.09477.pdf

https://arxiv.org/pdf/1509.06461.pdf

Hope this was useful.

UPDATE

There is a second option, you could define a custom loss function. Generally these functions only take two arguments, the predicted_y and the target_y, in your case, there is no target_y, so you could pass a dummy target_y and not use it inside the function (I assume that you could call your simulation process inside that function, and return the metric as the "loss"). Here are examples in PyTorch and Keras.

Keras: Make a custom loss function in keras

PyTorch:PyTorch custom loss function
