title: How can I exclude partitions when converting CSV to ORC using AWS Glue?
body: <p>I have a bunch of CSV files in S3 that I am trying to covert to ORC using an ETL job in AWS Glue. I have a crawler that crawls the directory containing the CSVs and generates a table. The table looks like this:</p>
  
  <pre><code>Column name | Data type | Partition key
  ---------------------------------------
  field1      | string    |
  field2      | string    |
  field3      | string    |
  partition_0 | string    | Partition (0)
  partition_1 | string    | Partition (1)
  </code></pre>
  
  <p>Next, I try to convert the CSVs into ORC files. Here is a similar ETL script to what I'm using:</p>
  
  <pre><code>args = getResolvedOptions(sys.argv, ['JOB_NAME', 'database', 'table_name', 'output_dir'])
  sc = SparkContext()
  glueContext = GlueContext(sc)
  spark = glueContext.spark_session
  job = Job(glueContext)
  job.init(args['JOB_NAME'], args)
  partition_predicate = '(partition_0 = "val1") AND (partition_1 = "val2")'
  datasource0 = glueContext.create_dynamic_frame.from_catalog(database = args['database'], table_name = args['table_name'], push_down_predicate = partition_predicate, transformation_ctx = "datasource0")
  final = glueContext.write_dynamic_frame.from_options(frame = datasource0, connection_type = "s3", connection_options = { "path": args['output_dir'] }, format = "orc")
  job.commit()
  </code></pre>
  
  <p>I have another crawler that crawls my output directory containing the ORC files. When it generates the table, it looks like this:</p>
  
  <pre><code>Column name | Data type | Partition key
  ---------------------------------------
  field1      | string    |
  field2      | string    |
  field3      | string    |
  partition_0 | string    |
  partition_1 | string    |
  partition_0 | string    | Partition (0)
  partition_1 | string    | Partition (1)
  </code></pre>
  
  <p>It looks like it considers the partitions to be fields in the ORC file (which they should not be). How can I modify my script so that the CSV to ORC conversion won't include the partition keys as schema columns?</p>