Parallel hyperparameter optimization with pytorch on a multi-gpu machine
I have access to a multi-gpu machine and I am running a grid search loop for parameter optimisation. I would like to know if I can distribute several iterations of the loop on multiple gpu at the same time, and if so how do I do it (what me mechanism? threading? how to gather the results if the loop execute asynchronously? etc.)
Thank you.

I'd suggest using Optuna to handle hyper-parameters search, which should in general perform better than grid search (you can still use it with grid sampling though). I have modified Optuna distributed example to use one GPU per process.

Create a training script like:

# optimize.py

import sys
import optuna
import your_model


DEVICE = 'cuda:' + sys.argv[1]

def objective(trial):
    hidden_size = trial.suggest_int('hidden_size', 8, 64, log=True)
    # define other hyperparameters

    return your_model.score(hidden_size=hidden_size, device=DEVICE)


if __name__ == '__main__':
    study = optuna.load_study(study_name='distributed-example', storage='sqlite:///example.db')
    study.optimize(objective, n_trials=100)


In terminal:

pip install optuna
optuna create-study --study-name &quot;distributed-example&quot; --storage &quot;sqlite:///example.db&quot;


Then for every GPU device:

python optimize.py 0
python optimize.py 1
...


Finally, best results can be easily discovered:

import optuna

study = optuna.create_study(study_name='distributed-example', storage='sqlite:///example.db', load_if_exists=True)

print(study.best_params)
print(study.best_value)

Or even visualized.
