title: SageMaker Managed Spot Training with Object Detection algorithm
body: <p>I'm trying to train an object detection model starting from an existing model using the new Managed Spot Training feature,  The paramters used when creating my Estimator are as follows:</p>
  
  <pre class="lang-py prettyprint-override"><code>od_model = sagemaker.estimator.Estimator(get_image_uri(sagemaker.Session().boto_region_name, 'object-detection', repo_version="latest"),
                                           Config['role'],
                                           train_instance_count = 1,
                                           train_instance_type = 'ml.p3.16xlarge',
                                           train_volume_size = 50,
                                           train_max_run = (48 * 60 * 60),
                                           train_use_spot_instances = True,
                                           train_max_wait = (72 * 60 * 60),
                                           input_mode = 'File',
                                           checkpoint_s3_uri = Config['train_checkpoint_uri'],
                                           output_path = Config['s3_output_location'],
                                           sagemaker_session = sagemaker.Session()
                                           )
  </code></pre>
  
  <p>(The references to <code>Config</code> in the above are a config data structure I'm using to extract/centralise some parameters)</p>
  
  <p>When I run the above, I get the following exception:</p>
  
  <blockquote>
    <p>botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreateTrainingJob operation: MaxWaitTimeInSeconds above 3600 is not supported for the given algorithm.</p>
  </blockquote>
  
  <p>If I change <code>train_max_wait</code> to 3600 I get this exception instead:</p>
  
  <blockquote>
    <p>botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreateTrainingJob operation: Invalid MaxWaitTimeInSeconds. It must be present and be greater than or equal to MaxRuntimeInSeconds</p>
  </blockquote>
  
  <p>However changing <code>max_run_time</code> to 3600 or less isn't going to work for me as I expect this model to take several days to train (large data set), in fact a single epoch takes more than an hour.</p>
  
  <p>The <a href="https://aws.amazon.com/blogs/aws/managed-spot-training-save-up-to-90-on-your-amazon-sagemaker-training-jobs/" rel="nofollow noreferrer">AWS blog post on Managed Spot Training</a> say that <code>MaxWaitTimeInSeconds</code> is limited to an 60 minutes for:</p>
  
  <blockquote>
    <p>For built-in algorithms and AWS Marketplace algorithms that don’t use checkpointing, we’re enforcing a maximum training time of 60 minutes (MaxWaitTimeInSeconds parameter).</p>
  </blockquote>
  
  <p>Earlier, the same blog post says:</p>
  
  <blockquote>
    <p>Built-in algorithms: computer vision algorithms support checkpointing (Object Detection, Semantic Segmentation, and very soon Image Classification).</p>
  </blockquote>
  
  <p>So I don't think it's that my algorithm doesn't support Checkpointing.  In fact that blog post uses object detection and max run times of 48 hours.  So I don't think it's an algorithm limitation.</p>
  
  <p>As you can see above, I've set up a S3 URL for the checkpoints.  The S3 bucket does exist, and the training container has access to it (it's the same bucket that the training data and model outputs are placed, and I had no problems with access to those before turning on spot training.</p>
  
  <p>My boto and sagemaker libraries are current versions:</p>
  
  <pre><code>boto3 (1.9.239)
  botocore (1.12.239)
  sagemaker (1.42.3)
  </code></pre>
  
  <p>As best I can tell from reading various docs, I've got everything set up correctly.  My use case is almost exactly what's described in the blog post linked above, but I'm using the SageMaker Python SDK instead of the console.</p>
  
  <p>I'd really like to try Managed Spot Training to save some money, as I have a very long training run coming up.  But limiting timeouts to an hour isn't going to work for my use case.  Any suggestions?</p>
  
  <p><strong>Update:</strong>  If I comment out the <code>train_use_spot_instances</code> and <code>train_max_wait</code> options to train on regular on-demand instances my training job is created successfully.  If I then try to use the console to clone the job and turn on Spot instances on the clone I get the same ValidationException.</p>