CI/CD with a GPU cluster
Since you're talking about CI/CD I presume you have the possibility to automate the model trainings in those configurations. Let's call the scripts able to do that train_model_config_A, train_model_config_B, etc.

Then you could have a wrapper script which checks an environment variable used to select which client environment you desire and invokes the corresponding train_model_config_&lt;blah&gt; script. Ideally translating the outcome of the training (whatever that is) into one or more results of the pass/fail type. Then such wrapper script can be integrated in a CI/CD pipeline like a custom test step/stage (or even a build one, if it produces any artifacts you might want to archive). Just like any test executed on a testbed incorporating some non-generic piece of hardware. In other words the CPU cluster makes no real difference.

You might not need to install the slave directly on the GPU cluster if the train_model_config_&lt;blah&gt; scripts (and thus the wrapper script as well) can be executed on some other hosts and remotely control the GPUs - you can then have the slave on some other host, leaving the GPU cluster free to only do its stuff.