Is there a relationship between learning rate and training set size?
I wonder why would you tune the learning rate (too much) at all? On many platforms you can include something like a learning rate scheduler. It will decrease the learing rate when no more learning progress is made. So you &quot;tune&quot; the learning rate &quot;on the fly&quot; while training.
With Keras this could look like:
early_stopping = EarlyStopping(monitor='val_loss', patience=50, mode='auto', restore_best_weights=True)
reduce_on_plateau = ReduceLROnPlateau(monitor=&quot;val_loss&quot;, factor=0.9, patience=20, cooldown=5, verbose=0)
checkpoint = ModelCheckpoint(&quot;xy.hdf5&quot;, monitor='val_loss', verbose=0, save_best_only=True, save_weights_only=True, mode='min')
...
model.fit(...,callbacks=[early_stopping, reduce_on_plateau, checkpoint]

The callbacks above introduce &quot;early stopping&quot; (to stop if no more progress is made), &quot;reduce on plateau&quot; (decrease learning rate when required), and &quot;checkpoint&quot; (save model results on each epoch).
Start with some reasonable standard value for the learning rate and decrease if necessary.