How to use AWS Sagemaker with newer version of Huggingface Estimator?
When trying to use Huggingface estimator on sagemaker, Run training on Amazon SageMaker e.g.
# create the Estimator
huggingface_estimator = HuggingFace(
        entry_point='train.py',
        source_dir='./scripts',
        instance_type='ml.p3.2xlarge',
        instance_count=1,
        role=role,
        transformers_version='4.17',
        pytorch_version='1.10',
        py_version='py38',
        hyperparameters = hyperparameters
)

When I tried to increase the version to transformers_version='4.24', it throws an error where the maximum version supported is 4.17.
How to use AWS Sagemaker with newer version of Huggingface Estimator?
There's a note on using newer version for inference on https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/9 but it looks like the way to use it for training with the Huggingface estimator is kind of complicated https://discuss.huggingface.co/t/huggingface-pytorch-versions-on-sagemaker/26315/5?u=alvations and it's not confirmed that the complicated steps can work.

You can use the Pytorch estimator and in your source directory place a requirements.txt with Transformers added to it. This will ensure 2 things

You can use higher version of pytorch 1.12 (current) compared to 1.10.2 in the huggingface estimator.
Install new version of HuggingFace Transformers library.

To achieve this you need to structure your source directory like this
scripts
/train.py
/requirements.txt
and pass the source_dir attribute to the pytorch estimator
pt_estimator = PyTorch(
entry_point=&quot;train.py&quot;,
source_dir=&quot;scripts&quot;,
role=sagemaker.get_execution_role(),

