title: Kafka Connect S3 Dynamic S3 Folder Structure Creation?
body: <p>I have <a href="https://docs.confluent.io/5.2.1/connect/managing/install.html#connect-install-connectors" rel="nofollow noreferrer">manually installed Confluent Kafka Connect S3</a> using the standalone method and not through Confluent's process or as part of the whole platform.</p>
  
  <p>I can successfully launch the connector from the command line with the command:</p>
  
  <pre><code>./kafka_2.11-2.1.0/bin/connect-standalone.sh connect.properties s3-sink.properties
  </code></pre>
  
  <p>Topic CDC offsets from <a href="https://aws.amazon.com/msk/" rel="nofollow noreferrer">AWS MSK</a> can be seen being consumed. No errors are thrown. However, in AWS S3, no folder structure is created for new data and no JSON data is stored.</p>
  
  <p><strong>Questions</strong></p>
  
  <ol>
  <li>Should the connector dynamically create the folder structure as it
  sees the first JSON packet for a topic? </li>
  <li>Other than configuring
  awscli credentials, connect.properties and s3-sink.properties are
  there any other settings that need to be set to properly connect to
  the S3 bucket? </li>
  <li>Recommendations on install documentation more
  comprehensive than the standalone docs on the Confluent website? (linked above)</li>
  </ol>
  
  <p><strong>connect.properties</strong></p>
  
  <blockquote>
    <p>bootstrap.servers=redacted:9092,redacted:9092,redacted:9092</p>
    
    <p>plugin.path=/plugins/kafka-connect-s3 key.converter=org.apache.kafka.connect.json.JsonConverter
    value.converter=org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable=false
    value.converter.schemas.enable=false
    internal.key.converter=org.apache.kafka.connect.json.JsonConverter
    internal.value.converter=org.apache.kafka.connect.json.JsonConverter
    internal.key.converter.schemas.enable=false
    internal.value.converter.schemas.enable=false
    offset.storage.file.filename=/tmp/connect.offsets</p>
  </blockquote>
  
  <p><strong>s3-sink.properties</strong></p>
  
  <blockquote>
    <p>name=s3-sink connector.class=io.confluent.connect.s3.S3SinkConnector
    tasks.max=1
    topics=database_schema_topic1,database_schema_topic2,database_schema_topic3
    s3.region=us-east-2 s3.bucket.name=databasekafka s3.part.size=5242880
    flush.size=1 storage.class=io.confluent.connect.s3.storage.S3Storage
    format.class=io.confluent.connect.s3.format.json.JsonFormat
    schema.generator.class=io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator
    partitioner.class=io.confluent.connect.storage.partitioner.DefaultPartitioner
    schema.compatibility=NONE</p>
  </blockquote>