title: Should I run forecast predictive model with AWS lambda or sagemaker?
body: <p>I've been reading some articles regarding this topic and have preliminary thoughts as what I should do with it, but still want to see if anyone can share comments if you have more experience with running machine learning on AWS. I was doing a project for a professor at school, and we decided to use AWS. I need to find a cost-effective and efficient way to deploy a forecasting model on it. </p>
  
  <p>What we want to achieve is:</p>
  
  <ul>
  <li>read the data from S3 bucket monthly (there will be new data coming in every month), </li>
  <li>run a few python files (.py) for custom-built packages and install dependencies (including the files, no more than 30kb), </li>
  <li>produce predicted results into a file back in S3 (JSON or CSV works), or push to other endpoints (most likely to be some BI tools - tableau etc.) - but really this step can be flexible (not web for sure) </li>
  </ul>
  
  <p><strong>First thought I have is AWS sagemaker</strong>. However, we'll be using "fb prophet" model to predict the results, and we built a customized package to use in the model, therefore, I don't think the notebook instance is gonna help us. (Please correct me if I'm wrong) My understanding is that sagemaker is a environment to build and train the model, but we already built and trained the model. Plus, we won't be using AWS pre-built models anyways.</p>
  
  <p>Another thing is if we want to use custom-built package, we will need to create container image, and I've never done that before, not sure about the efforts to do that.</p>
  
  <p><strong>2nd option is to create multiple lambda functions</strong></p>
  
  <ul>
  <li><p>one that triggers to run the python scripts from S3 bucket (2-3 .py files) every time a new file is imported into S3 bucket, which will happen monthly.</p></li>
  <li><p>one that trigger after the python scripts are done running and produce results and save into S3 bucket.</p></li>
  </ul>
  
  <p>3rd option will combine both options:
   - Use lambda function to trigger the implementation on the python scripts in S3 bucket when the new file comes in.
   - Push the result using sagemaker endpoint, which means we host the model on sagemaker and deploy from there.</p>
  
  <p>I am still not entirely sure how to put pre-built model and python scripts onto sagemaker instance and host from there.</p>
  
  <p>I'm hoping whoever has more experience with AWS service can help give me some guidance, in terms of more cost-effective and efficient way to run model.</p>
  
  <p>Thank you!! </p>