Results reproducibility using Pytorch and Optuna for DNN
Finally, I was able to find the reason for improper reproducibility. In my code I used two different objective functions; def train(trial) and def layer(trial). I pivoted the second objective function into the train(trial). Also, specifying a manual seed is also important. Anyway there will be slight deviations of 0.0001%.
I have found the optimal results for 7 hyperparameter namely:

Number of layers,
Node size,
Activation functions,
learning rate,
momentum,
batch size,
optimizer

Using Optuna multiobjective optimization. I minimized the training and validation loss as my objectives. Since the number of tuning parameters are more I reduced the number of epoch per trail as 50. Then I got the best parameters, post Optuna optimization. I increased the epoch size and build the same model with torch.manual_seed. But the results obtained after the same 50th epoch is different from what I got in the Optuna results.
What is the reason am I missing anything? I want to reproduce the same results for the same condition!