Training with many CPU cores doesn't improve performance I ran my job on a computing cluster: first with 1 node / 4 cores, then with 2 nodes / 32 cores. But the training time is pretty much exactly the same for both of them! 67 seconds per step.

I am trying to fine-tune GPT-2 for my text dataset (chat logs).

What can I do to get a performance increase corresponding to the increase in processing power with CPUs?
Have you done any analysis as to what steps in your procedure consume most time? For example, it may be that GPU operations are responsible for most of the time. Then, of course, adding CPU cores will not give any benefit. And have you looked at CPU statistics, i.e. is the full capacity used?