title: How do I implement a PyTorch Dataset for use with AWS SageMaker?
body: <p>I have implemented a PyTorch <code>Dataset</code> that works locally (on my own desktop), but when executed on AWS SageMaker, it breaks. My <code>Dataset</code> implementation is as follows.</p>
  
  <pre><code>class ImageDataset(Dataset):
      def __init__(self, path='./images', transform=None):
          self.path = path
          self.files = [join(path, f) for f in listdir(path) if isfile(join(path, f)) and f.endswith('.jpg')]
          self.transform = transform
          if transform is None:
              self.transform = transforms.Compose([
                  transforms.Resize((128, 128)),
                  transforms.ToTensor(),
                  transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
              ])
  
      def __len__(self):
          return len(files)
  
      def __getitem__(self, idx):
          img_name = self.files[idx]
  
          # we may infer the label from the filename
          dash_idx = img_name.rfind('-')
          dot_idx = img_name.rfind('.')
          label = int(img_name[dash_idx + 1:dot_idx])
  
          image = Image.open(img_name)
  
          if self.transform:
              image = self.transform(image)
  
          return image, label
  </code></pre>
  
  <p>I am following this <a href="https://github.com/aws/sagemaker-python-sdk/tree/master/src/sagemaker/pytorch" rel="noreferrer">example</a> and this <a href="https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/pytorch_cnn_cifar10/pytorch_local_mode_cifar10.ipynb" rel="noreferrer">one too</a>, and I run the <code>estimator</code> as follows.</p>
  
  <pre><code>inputs = {
   'train': 'file://images',
   'eval': 'file://images'
  }
  estimator = PyTorch(entry_point='pytorch-train.py',
                              role=role,
                              framework_version='1.0.0',
                              train_instance_count=1,
                              train_instance_type=instance_type)
  estimator.fit(inputs)
  </code></pre>
  
  <p>I get the following error.</p>
  
  <blockquote>
    <p>FileNotFoundError: [Errno 2] No such file or directory: './images'</p>
  </blockquote>
  
  <p>In the example that I am following, they upload the CFAIR dataset (which is downloaded locally) to S3.</p>
  
  <pre><code>inputs = sagemaker_session.upload_data(path='data', bucket=bucket, key_prefix='data/cifar10')
  </code></pre>
  
  <p>If I take a peek at <code>inputs</code>, it is just a string literal <code>s3://sagemaker-us-east-3-184838577132/data/cifar10</code>. The code to create a <code>Dataset</code> and a <code>DataLoader</code> is shown <a href="https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/pytorch_mnist/mnist.py#L41" rel="noreferrer">here</a>, which does not help unless I track down the source and step through the logic.</p>
  
  <p>I think what needs to happen inside my <code>ImageDataset</code> is to supply the <code>S3</code> path and use the <code>AWS CLI</code> or something to query the files and acquire their content. I do not think the <code>AWS CLI</code> is the right approach as this relies on the console and I will have to execute some sub-process commands and then parse through. </p>
  
  <p>There must be a recipe or something to create a custom <code>Dataset</code> backed by <code>S3</code> files, right?</p>