title: How to load streaming data from Amazon SQS?
body: <p>I use Spark 2.2.0.</p>
  
  <p>How can I feed Amazon SQS stream to spark structured stream using pyspark?</p>
  
  <p><a href="https://stackoverflow.com/questions/27123006/feeding-apache-spark-streaming-from-amazon-sqs">This</a> question tries to answer it for a non structured streaming and for scala by creating a custom receiver.<br>
  Is something similar possible in pyspark? </p>
  
  <pre><code>spark.readStream \
     .format("s3-sqs") \
     .option("fileFormat", "json") \
     .option("queueUrl", ...) \
     .schema(...) \
     .load()
  </code></pre>
  
  <p>According to <a href="https://docs.databricks.com/spark/latest/structured-streaming/sqs.html#" rel="noreferrer">Databricks</a> above receiver can be used for <strong>S3-SQS</strong> file source. However, for only SQS how may one approach. </p>
  
  <p>I tried understanding from <a href="http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html#" rel="noreferrer">AWS-SQS-Receive_Message</a> to receive message. However, how to directly send stream to spark streaming was not clear.</p>