
Workflow on Clusters
The high level idea is to have a cluster that has a head node which controls the compute nodes. The actual training job runs on the compute nodes. Overall steps to run a training job on a cluster are as follows:

Set up an AWS ParallelCuster (we discuss below)
Connect to the head node, and import the training code/ setup the environment.
Pull the data and place it in a shared folder that compute nodes can access (FSx Lustre drive).
Run the training job using a job scheduler (in this case Slurm).

Setup AWS ParallelCuster
To setup AWS ParallelCluster,

Deploy a network stack. This step is optional since you could use your account default VPC and let AWS ParallelCluster create your subnets and security groups. However, we prefer to compartmentalize our desired network infrastructure and do this deployment via a CloudFormation stack.
