title: Conflicting partition column names detected Pyspark Databricks
body: <p>I'm trying to read a csv file with pyspark in databricks.
  marketingStartDate is this format <code>yyyyMMdd</code> and <code>lastweek = marketingStartDate -7days</code></p>
  <pre><code>readFactToDataFrame('Facts', 
  'Fact.csv',startDate=str(dateBefore),
             endDate=str(marketingStartDate),
             inferSchema=False)
  </code></pre>
  <p>I am getting this error message. Do you have any idea where the problem lies ?</p>
  <p><a href="https://i.stack.imgur.com/PX8xZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/PX8xZ.png" alt="enter image description here" /></a></p>
  <pre><code>df = spark.read.format(fileFormat).options(header=True, inferSchema=
  inferSchema, delimiter = columnDelimiter).load(URL).filter(&quot;Year *
  10000 + Month * 100 + Day = &quot;+str(startDate) + &quot; AND Year * 10000 +
  Month * 100 + Day &lt;=&quot; + str(endDate)) 37 elif fileFormat == &quot;json&quot;: 38
  df =
  spark.read.format(fileFormat).options(multiline=True).load(URL).filter(&quot;Year
  * 10000 + Month * 100 + Day = &quot;+str(startDate) + &quot; AND Year * 10000 + Month * 100 + Day &lt;=&quot; + str(endDate))
  /databricks/spark/python/pyspark/sql/readwriter.py in load(self, path,
  format, schema, **options) 164 self.options(**options) 165 if
  isinstance(path, basestring): -- 166 return
  self._df(self._jreader.load(path)) 167 elif path is not None: 168 if
  type(path) != list:
  /databricks/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py
  in __call__(self, *args) 1255 answer =
  self.gateway_client.send_command(command) 1256 return_value =
  get_return_value( - 1257 answer, self.gateway_client, self.target_id,
  self.name) 1258 1259 for temp_arg in temp_args:
  /databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw) 61 def
  deco(*a, **kw): 62 try: --- 63 return f(*a, **kw) 64 except
  py4j.protocol.Py4JJavaError as e: 65 s = e.java_exception.toString()
  /databricks/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in
  get_return_value(answer, gateway_client, target_id, name) 326 raise
  Py4JJavaError( 327 &quot;An error occurred while calling {0}{1}{2}.\n&quot;. --
  328 format(target_id, &quot;.&quot;, name), value) 329 else: 330 raise
  Py4JError( Py4JJavaError: An error occurred while calling o2301.load.
  : java.lang.AssertionError: assertion failed: Conflicting partition
  column names detected: Partition column name list #0: Year, Month, Day
  Partition column name list #1: Year, Month For partitioned table
  directories, data files should only live in leaf directories. And
  directories at the same level should have the same partition column
  name. Please check the following directories for unexpected files or
  inconsistent partition column names:
  dbfs:/mnt/DL/Facts/Year=2020/Month=08
  dbfs:/mnt/DL/Facts/Year=2020/Month=03/Day=16
  dbfs:/mnt/DL/Facts/Year=2019/Month=09/Day=27
  dbfs:/mnt/DL/Facts/Year=2019/Month=09/Day=02
  dbfs:/mnt/DL/Facts/Year=2020/Month=08/Day=01
  dbfs:/mnt/DL/Facts/Year=2020/Month=03/Day=09
  dbfs:/mnt/DL/Facts/Year=2020/Month=02/Day=26
  dbfs:/mnt/DL/Facts/Year=2020/Month=08/Day=10
  dbfs:/mnt/DL/Facts/Year=2019/Month=09/Day=12
  dbfs:/mnt/DL/Facts/Year=2019/Month=10/Day=12
  dbfs:/mnt/DL/Facts/Year=2020/Month=03/Day=24
  dbfs:/mnt/DL/Facts/Year=2019/Month=09/Day=05
  dbfs:/mnt/DL/Facts/Year=2020/Month=03/Day=13
  dbfs:/mnt/DL/Facts/Year=2019/Month=10/Day=27
  dbfs:/mnt/DL/Facts/Year=2019/Month=09/Day=16
  dbfs:/mnt/DL/Facts/Year=2020/Month=02/Day=20
  dbfs:/mnt/DL/Facts/Year=2020/Month=03/Day=31 at
  scala.Predef$.assert(Predef.scala:170) at
  org.apache.spark.sql.execution.datasources.PartitioningUtils$.resolvePartitions(PartitioningUtils.scala:396)
  at
  org.apache.spark.sql.execution.datasources.PartitioningUtils$.parsePartitions(PartitioningUtils.scala:197)
  at
  org.apache.spark.sql.execution.datasources.PartitioningUtils$.parsePartitions(PartitioningUtils.scala:132)
  </code></pre>