title: [Structured Streaming]: Structured Streaming into Redshift sink
body: <p>Is it possible to write the Dataframe backed by Kafka Streaming source into AWS Redshift, we have in the past used <a href="https://github.com/databricks/spark-redshift" rel="nofollow noreferrer">spark-redshift</a> to write into Redshift, but I presume it will not work with <code>DataFrame##writeStream</code>. Also writing with JDBC connector with <code>ForeachWriter</code> is also may not be a good idea given the way Redshift works. </p>
  
  <p>One possible approach that I have come across from <a href="https://engineeringblog.yelp.com/2016/10/redshift-connector.html" rel="nofollow noreferrer">Yelp blog</a> is to write the files into S3 and then invoke <a href="https://docs.aws.amazon.com/redshift/latest/dg/r_COPY.html" rel="nofollow noreferrer">Redshift <code>COPY</code></a> with a Manifest file having the S3 Object path, in case of Structured Streaming, how can I control the files into which I write to S3? and also have a separate trigger to create a manifest file after writing say 5 files into S3. </p>
  
  <p>Any other possible solution is also appreciated. Thanks in advance.</p>