Use SageMaker Pytorch image for training
I am trying to containerize the training process for a fine tuned BERT model and run it on SageMaker. I was planning to use the pre-built SageMaker Pytorch GPU containers (https://aws.amazon.com/releasenotes/available-deep-learning-containers-images/) as my starting point but I am having issues pulling the images during my build process.
My Dockerfile looks like this:
# SageMaker PyTorch image
FROM 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:1.5.0-gpu-py36-cu101-ubuntu16.04


ENV PATH=&quot;/opt/ml/code:${PATH}&quot;

# /opt/ml and all subdirectories are utilized by SageMaker, we use the /code subdirectory to store our user code.
COPY /bert /opt/ml/code

# this environment variable is used by the SageMaker PyTorch container to determine our user code directory.
ENV SAGEMAKER_SUBMIT_DIRECTORY /opt/ml/code

# this environment variable is used by the SageMaker PyTorch container to determine our program entry point
# for training and serving.
# For more information: https://github.com/aws/sagemaker-pytorch-container
ENV SAGEMAKER_PROGRAM bert/train

My build_and_push script:
#!/usr/bin/env bash

# This script shows how to build the Docker image and push it to ECR to be ready for use
# by SageMaker.

# The argument to this script is the image name. This will be used as the image on the local
# machine and combined with the account and region to form the repository name for ECR.
IMAGE=&quot;my-bert&quot;

# parameters
PY_VERSION=&quot;py36&quot;

# Get the account number associated with the current IAM credentials
account=$(aws sts get-caller-identity --query Account --output text)

if [ $? -ne 0 ]
then
    exit 255
fi

chmod +x bert/train

# Get the region defined in the current configuration (default to us-west-2 if none defined)
region=$(aws configure get region)
region=${region:-us-east-2}

# If the repository doesn't exist in ECR, create it.
aws ecr describe-repositories --repository-names ${IMAGE} || aws ecr create-repository --repository-name ${IMAGE}

echo &quot;---&gt; repository done..&quot;
# Get the login command from ECR and execute it directly
aws ecr get-login-password --region $region | docker login --username AWS --password-stdin $account.dkr.ecr.$region.amazonaws.com
echo &quot;---&gt; logged in to account ecr..&quot;

# Get the login command from ECR in order to pull down the SageMaker PyTorch image
# aws ecr get-login-password --region $region | docker login --username AWS --password-stdin 763104351884.dkr.ecr.us-east-1.amazonaws.com
# echo &quot;---&gt; logged in to pytorch ecr..&quot;

echo &quot;Building image with arch=gpu, region=${region}&quot;
TAG=&quot;gpu-${PY_VERSION}&quot;
FULLNAME=&quot;${account}.dkr.ecr.${region}.amazonaws.com/${IMAGE}:${TAG}&quot;
docker build -t ${IMAGE}:${TAG} --build-arg ARCH=&quot;$arch&quot; -f &quot;Dockerfile&quot; .
docker tag ${IMAGE}:${TAG} ${FULLNAME}
docker push ${FULLNAME}


I get the following message during the push and the sagemaker pytorch image is not pulled:
Get https://763104351884.dkr.ecr.us-east-1.amazonaws.com/v2/pytorch-training/manifests/1.5.0-gpu-py36-cu101-ubuntu16.04: no basic auth credentials

Please let me know if this is the correct way to use a pre-built SageMaker image and what I could do to fix this error.

You should run something like this, before running the docker build:
aws ecr get-login-password --region ${region} | docker login --username AWS --password-stdin 763104351884.dkr.ecr.${region}.amazonaws.com

