title: How to to call a python script to execute as part of an EC2 cluster workflow via aws cli?
body: <p>I am calling for the creation of an ec2 instance like so:</p>
  
  <pre><code>aws2 emr create-cluster \
  --name "Spark cluster with step" \
  --release-label emr-5.24.1 \
  --applications Name=Spark \
  --log-uri s3://boris-log-bucket/logs/ \
  --ec2-attributes KeyName=boris-aws,EmrManagedMasterSecurityGroup=sg-...,EmrManagedSlaveSecurityGroup=sg-... \
  --instance-type m5.xlarge \
  --instance-count 1 \
  --bootstrap-actions Path=s3://boris-set-up/bootstrap_file2.sh \
  --steps Type=Spark,Name="Spark job",ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,--master,yarn] \
  --use-default-roles \
  --no-auto-terminate
  </code></pre>
  
  <p>The contents of <code>bootstrap_file2.sh</code> include:</p>
  
  <pre><code>aws s3 sync s3://boris-scripts/ /home/hadoop/
  </code></pre>
  
  <p>Which will copy a python script into the ec2 instance.</p>
  
  <p>Now I want to have something along the lines of "Once bootstrap finishes, run the python script"</p>
  
  <p>I've tried two things:
  1) I added the line <code>sudo python3 my_script.py</code> into the bootstrap bash script and it fails
  2) I modified the steps to be <code>--steps Type=Spark,Name="Spark job",ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,--master,yarn,s3://boris-scripts/my_script.py]</code> and that also fails</p>
  
  <p>My script <code>my_script.py</code> tries to grab data from Wikipedia and write it to /home/hadoop/ as a csv file</p>