title: Delta Lake Spark compaction after merge operation gives 'DeltaTable' object has no attribute '_get_object_id' error
body: <p>I am doing delta-lake merge operation using python api and pyspark . After doing the merge operation I call the compaction operation but the compaction gives the following error:</p>
  
  <p>Error:</p>
  
  <pre><code>  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 170, in load
    File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1248, in __call__
    File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1212, in _build_args
    File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1199, in _get_args
    File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_collections.py", line 501, in convert
    File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1248, in __call__
    File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1218, in _build_args
    File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1218, in &lt;listcomp&gt;
    File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 298, in get_command_part
  AttributeError: 'DeltaTable' object has no attribute '_get_object_id'
  </code></pre>
  
  <p>Code</p>
  
  <pre><code>delta_table = "delta_lake_path"
  
  df = spark.read.csv("s3n://input_file.csv",header=True)
  
  delta_table = DeltaTable.forPath(spark, delta_table)
  
  delta_table.merge(df, "df.id = delta_table.id" ).whenNotMatchedInsertAll().execute()
  
  #compaction
  spark.read.format("delta").load(delta_table).repartition(1).write.option("dataChange",
  "False").format("delta").mode("overwrite").save(delta_table)
  </code></pre>
  
  <p>Can anyone suggest me why the spark session is not able to create another delta table instance .
  I need to perform both merge and compaction in the same script since I want to run the compaction only on the partitions in which the merge is performed . The partitions are derived from the unique values present in the dataframe df created from input_file.csv</p>