# How to tune optimizer hyperparameters with Trainer.hyperparameter_search

Hi, I have a working hyperparameter tuning setup that looks something like the following:

``````training_args = TrainingArguments(
output_dir="test_trainer",
save_strategy="epoch",
save_total_limit=1,
evaluation_strategy="steps",
num_train_epochs=epochs,
per_device_train_batch_size=batch_size,
per_device_eval_batch_size=batch_size,
lr_scheduler_type="cosine",
warmup_steps=500,
optim=optimizer)

trainer = Trainer(
model=None,
model_init=model_init,
args=training_args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
compute_metrics=compute_cls_metrics,
tokenizer=tokenizer,
)

def ray_hp_space(trial):
return {
"learning_rate": tune.loguniform(1e-6, 1e-2),
"per_device_train_batch_size":
tune.choice([16, 32, 64, 128]),
}

best_trial = trainer.hyperparameter_search(
direction="maximize",
backend="ray",
hp_space=ray_hp_space,
n_trials=tune_num_trials,
compute_objective=lambda metrics: metrics['eval_accuracy'],
search_alg=HyperOptSearch(metric="objective", mode="max"),
scheduler=ASHAScheduler(metric="objective", mode="max")
)
``````

This is based on the example code from the docs here. However, I’m noticing that the `learning_rate` hyperparameter in the docs doesn’t seem to be used anywhere – it seems like the `learning_rate` is not actually being tuned at all by this example, which is instead passing the dummy trial config that is not consumed anywhere (since `model_init` is the only place where the trial parameters are provided, and `model_init` does not use the learning rate).

Optimization hyperparameters are probably the most commonly-tuned parameters. So, I am wondering, how can we actually tune the optimizer hyperparameters? This would include other parameters besides learning rate, such as weight_decay and betas for Adam.

Even subclassing Trainer and overriding `create_optimizer_and_scheduler()` doesn’t seem like it would accomplish this, because `create_optimizer_and_scheduler()` doesn’t have access to the hyperparameters being used for a given tuning run.

Thanks for any suggestions you can provide! I would be happy to contribute some updates to the docs to demonstrate this if that functionality exists but is not currently demonstrated in the documentation.
{"ft_lang_label":"__label__en","ft_lang_prob":0.7905801,"math_prob":0.74370146,"size":2175,"snap":"2023-14-2023-23","text_gpt3_token_len":467,"char_repetition_ratio":0.10916628,"word_repetition_ratio":0.0,"special_character_ratio":0.21241379,"punctuation_ratio":0.16463415,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9946833,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2023-03-28T21:38:54Z\",\"WARC-Record-ID\":\"<urn:uuid:7c63da07-2126-403b-931c-c5a0005b3018>\",\"Content-Length\":\"19214\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:e22796fd-d20a-4fd4-9cbf-15d29e696eb8>\",\"WARC-Concurrent-To\":\"<urn:uuid:953aff85-e19a-40f0-8917-8a00e65d0fd8>\",\"WARC-IP-Address\":\"184.105.176.45\",\"WARC-Target-URI\":\"https://discuss.huggingface.co/t/how-to-tune-optimizer-hyperparameters-with-trainer-hyperparameter-search/33191\",\"WARC-Payload-Digest\":\"sha1:V7CATL3UA32ITWBPCZXVDALFSX62T3E7\",\"WARC-Block-Digest\":\"sha1:BHYXAHJIR5MRW3PBYHZQOPMZ23NMLLMD\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2023/CC-MAIN-2023-14/CC-MAIN-2023-14_segments_1679296948871.42_warc_CC-MAIN-20230328201715-20230328231715-00774.warc.gz\"}"}