title: Multi GPU training for Transformers with different GPUs
body: <p>I want to fine tune a GPT-2 model using Huggingface’s Transformers. Preferably the medium model but large if possible. Currently, I have a RTX 2080 Ti with 11GB of memory and I can train the small model just fine.</p>
  
  <p>My question is: will I run into any issues if I added an old Tesla K80 (24GB) to my machine and distributed the training? I cannot find information about using different capacity GPUs during training and issues I could run into.</p>
  
  <p>Will my model size limit essentially be sum of all available GPU memory? (35GB?)</p>
  
  <p>I’m not interested in doing this in AWS.</p>