title: How to return output from AWS Glue jobs back to the calling Step Function workflow?
body: <p>AWS Step Functions allow calling AWS Glue jobs, as described here:
  <a href="https://docs.aws.amazon.com/step-functions/latest/dg/connect-glue.html" rel="noreferrer">https://docs.aws.amazon.com/step-functions/latest/dg/connect-glue.html</a></p>
  
  <p>I want to run the job and (after saving the results to S3) return some metadata produced during the job (like row count or number of filtered rows) back to the Step function flow.</p>
  
  <p>We can pass parameters from Step functions to the Glue job like this:</p>
  
  <pre><code>              "RunGlueJob": {
                  "Type": "Task",
                  "Resource": "arn:aws:states:::glue:startJobRun.sync",
                  "Parameters": {
                    "JobName": "MyJobName",
                    "Arguments": {
                      "--param1.$": "$.param1",
                      "--param2.$": "$.param2"
                    }
                  },
                  "Next": "NextState"
                },
  </code></pre>
  
  <p>But how can the Glue job return output back to the Step Function workflow? I tried just returning a String from the main() function inside the (Scala) Glue job, but it doesn't show up in JSON returned to the step function flow:</p>
  
  <pre><code>{
        "AllocatedCapacity": 3,
        "Arguments": {
          "--param1.$": "$.param1",
          "--param2.$": "$.param2"
        },
        "Attempt": 0,
        "CompletedOn": 1570114802442,
        "ExecutionTime": 39,
        "GlueVersion": "0.9",
        "Id": "jr_some_id",
        "JobName": "MyJobName",
        "JobRunState": "SUCCEEDED",
        "LastModifiedOn": 1570114802442,
        "LogGroupName": "/aws-glue/jobs",
        "MaxCapacity": 3,
        "PredecessorRuns": [],
        "StartedOn": 1570114746138,
        "Timeout": 2880
      }
  </code></pre>
  
  <p>I cannot find any documentation on this, so it might be that this is simply not possible. However, returning values from Lambdas works just fine and shows up normally inside the Step function workflow.</p>