title: How do I interact with Python code on EC2 instance via API-GW?
body: <p>I want to facilitate URI-based interaction with Python programs on the login node of a cfncluster.  The programs will perform various time-intensive (minutes to hours) operations, mostly linked through S3 objects produced and consumed.</p>
  
  <p>I have prototyped smaller scale scenarios using Lambda functions and API GW to provide asynchronous endpoints where the client sends JSON to the URI endpoint, which sends it to the Lambda function, the Lambda function comes up with a unique S3 object name and immediately returns that (in JSON) to the client, then goes to work producing stuff in that S3 object (within the time and space limitations of Lambda functions).  The client polls until the object is available, often calling another Lambda function to consume that S3 object.</p>
  
  <p>Now, I want to facilitate the same thing where the work being done is huge (large scale atmospheric models), which will require the launching of a cfncluster (for parallel computing) with lots of big software packages.  The best approach I can come up with so far is to launch Lambda functions via API-GW, then have them start asynchronous processes (launching, status-checking, killing) on the cfncluster.  Although I could have the Lambda functions interact with the cfncluster processes via ssh, I would prefer to avoid that.  These processes will generally come from running Python code.</p>
  
  <p>I've read of the AWS "<a href="https://aws.amazon.com/blogs/aws/new-ec2-run-command-remote-instance-management-at-scale/" rel="nofollow noreferrer">EC2 Run Command</a>" for performing admin tasks, and my vague understanding is that</p>
  
  <ul>
  <li>I can run this "EC2 Run Command" from a Lambda function, meaning I could pass JSON through API GW to the Lambda function and have that create the "run" command on the EC2 instance.</li>
  <li>The "EC2 Run Command" will write <em>stdout</em> to an S3 object, so "maybe" I can set up my EC2 processes to write out the JSON I would want in a response, and have other Lambda functions grab those in status requests.</li>
  </ul>
  
  <p>This, of course, doesn't seem too easy, but it seems do-able, and to the best of my knowledge just might represent the "state of the art" for accomplishing web-based control of large-scale scientific models in the cloud.  Am I correct in thinking that this is probably "state of the art" for what I want to do?  AWS constantly adds new services.  Have I missed something?</p>
  
  <p>I have also contemplated (and prototyped on a small scale) the use of a CherryPy server on the EC2 instance, but that has its own complexities and drawbacks. </p>