Architecture for Machine Learning resource management
It depends on how the cluster (i.e., compute resources) are being managed. For example, Apache Mesos is an open-source cluster management tool. 

One established option is Horovod. It is designed to manage workloads for 
TensorFlow and Keras across multiple cores in a single machine or across multiple machines.