title: Couldn't create model.tar.gz file while training scikit learn model in AWS Sagemaker
body: <p>I want to create an endpoint for scikit logistic regression in AWS Sagemaker. I have a <code>train.py</code> file which contains training code for scikit sagemaker. </p>
  
  <pre><code>import subprocess as sb
  import pandas as pd
  import numpy as np
  import pickle,json
  import sys
  
  def install(package):
      sb.call([sys.executable, "-m", "pip", "install", package])
  
  install('s3fs')
  
  import argparse
  import os
  
  if __name__ =='__main__':
      parser = argparse.ArgumentParser()
  
      # hyperparameters sent by the client are passed as command-line arguments to the script.
      parser.add_argument('--solver', type=str, default='liblinear')
  
      # Data, model, and output directories
      parser.add_argument('--output_data_dir', type=str, default=os.environ.get('SM_OUTPUT_DIR'))
      parser.add_argument('--model_dir', type=str, default=os.environ.get('SM_MODEL_DIR'))
      parser.add_argument('--train', type=str, default=os.environ.get('SM_CHANNEL_TRAIN'))
  
      args, _ = parser.parse_known_args()
  
      # ... load from args.train and args.test, train a model, write model to args.model_dir.
  
      input_files = [ os.path.join(args.train, file) for file in os.listdir(args.train) ]
      if len(input_files) == 0:
          raise ValueError(('There are no files in {}.\n' +
                            'This usually indicates that the channel ({}) was incorrectly specified,\n' +
                            'the data specification in S3 was incorrectly specified or the role specified\n' +
                            'does not have permission to access the data.').format(args.train, "train"))
  
      raw_data = [ pd.read_csv(file, header=None, engine="python") for file in input_files ]
  
      df = pd.concat(raw_data)
  
      y = df.iloc[:,0]
      X = df.iloc[:,1:]
  
  
  
      solver = args.solver
      from sklearn.linear_model import LogisticRegression
      lr = LogisticRegression(solver=solver).fit(X, y)
  from sklearn.externals import joblib
  def model_fn(model_dir):
      lr = joblib.dump(lr, "model.joblib")
      return lr
  </code></pre>
  
  <p>In my sagemaker notebook I ran the following code</p>
  
  <pre><code>import os
  import boto3
  import re
  import copy
  import time
  from time import gmtime, strftime
  from sagemaker import get_execution_role
  import sagemaker
  
  role = get_execution_role()
  
  region = boto3.Session().region_name
  
  bucket=&lt;bucket&gt; # Replace with your s3 bucket name
  prefix = &lt;prefix&gt;
  
  output_path = 's3://{}/{}/{}'.format(bucket, prefix,'output_data_dir')
  train_data = 's3://{}/{}/{}'.format(bucket, prefix, 'train')
  train_channel = sagemaker.session.s3_input(train_data, content_type='text/csv')
  
  from sagemaker.sklearn.estimator import SKLearn
  sklearn = SKLearn(
      entry_point='train.py',
      train_instance_type="ml.m4.xlarge",
      role=role,output_path = output_path,
      sagemaker_session=sagemaker.Session(),
      hyperparameters={'solver':'liblinear'})
  </code></pre>
  
  <p>I'm fitting my model here</p>
  
  <pre><code>sklearn.fit({'train': train_channel})
  </code></pre>
  
  <p>Now, for creating endpoint,</p>
  
  <pre><code>from sagemaker.predictor import csv_serializer
  predictor = sklearn.deploy(1, 'ml.m4.xlarge')
  </code></pre>
  
  <p>While trying to create endpoint, it is throwing </p>
  
  <pre><code>ClientError: An error occurred (ValidationException) when calling the CreateModel operation: Could not find model data at s3://&lt;bucket&gt;/&lt;prefix&gt;/output_data_dir/sagemaker-scikit-learn-x-y-z-000/output/model.tar.gz.
  </code></pre>
  
  <p>I checked my S3 bucket. Inside my <code>output_data_dir</code> there is <code>sagemaker-scikit-learn-x-y-z-000</code> dir which has <code>debug-output\training_job_end.ts</code> file. An additional directory got created outside my <code>&lt;prefix&gt;</code> folder with name <code>sagemaker-scikit-learn-x-y-z-000</code> that has <code>source\sourcedir.tar.gz</code> file. Generally whenever I trained my models with sagemaker built-in algorithms, <code>output_data_dir\sagemaker-scikit-learn-x-y-z-000\output\model.tar.gz</code> kind of files get created. Can someone please tell me where my scikit model got stored, how to push <code>source\sourcedir.tar.gz</code> inside my prefix code without having doing it manually and how to see contents of <code>sourcedir.tar.gz</code>?</p>
  
  <p>Edit: I elaborated the question regarding <code>prefix</code>. Whenever I run <code>sklearn.fit()</code>, two files with same name <code>sagemaker-scikit-learn-x-y-z-000</code> are getting created in my S3 bucket. One created inside my <code>&lt;bucket&gt;/&lt;prefix&gt;/output_data_dir/sagemaker-scikit-learn-x-y-z-000/debug-output/training_job_end.ts</code> and other file is created in <code>&lt;bucket&gt;/sagemaker-scikit-learn-x-y-z-000/source/sourcedir.tar.gz</code>. Why is the second file not created inside my <code>&lt;prefix&gt;</code> like the first one? What is contained in <code>sourcedir.tar.gz</code> file?</p>