Training with many CPU cores doesn't improve performance
Have you done any analysis as to what steps in your procedure consume most time? For example, it may be that GPU operations are responsible for most of the time. Then, of course, adding CPU cores will not give any benefit. And have you looked at CPU statistics, i.e. is the full capacity used?