title: How to execute EMR step that loads more scripts from s3?
body: <p>I want to execute a shell script as a step on EMR that loads a tarball, unzips it and runs the script inside. I chose this setup to stay as vendor-agnostic as possible.
  My script is</p>
  
  <pre><code>#!/bin/sh
  aws s3 cp s3://path_to_my_bucket/name_of.tar.gz .
  tar -xzf name_of.tar.gz
  . main_script.sh
  </code></pre>
  
  <p>Where <code>main_script.sh</code> is part of the tarball along with a number of other packages, scripts and config files.</p>
  
  <p>If I run this script as a Hadoop user on the master node, everything works as intended. Added as a step via the <a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hadoop-script.html" rel="nofollow noreferrer">command-runner.jar</a>, I get errors, no matter what I try.</p>
  
  <p>What I tried so far (and the errors):</p>
  
  <ul>
  <li>running the script as above (file not found "main_script.sh")</li>
  <li>hardcoding the path to be the Hadoop users home directory (permission denied on <code>main_script.sh</code>)</li>
  <li>dynamically getting the path where script lives (using <a href="https://stackoverflow.com/questions/59895/getting-the-source-directory-of-a-bash-script-from-within">this</a>) and giving this path as an argument for the <code>tar -C</code> option and invoking <code>main_script.sh</code> explicitly from this path (another permission denied on <code>main_script.sh</code>)</li>
  </ul>
  
  <p><strong>What is the proper way of loading a bash script into the master node and executing it?</strong></p>
  
  <p>As a bonus, I am wondering why the <code>command-runner.jar</code> is set up so different from the spark step, which runs as the Hadoop user in the Hadoop user directory.</p>