title: How to compare the headers of the df with other df headers using Fuzzy Matching in pyspark?
body: <p>I have created 3 dataframes by  executing following code. 
  sample.csv </p>
  
  <pre><code>id|code|name|Lname|mname
  2|AA|BB|CC|DD|
  </code></pre>
  
  <p>sample1.csv</p>
  
  <pre><code>id|code|name|Lname|mname
  1|A|B|C|D|
  </code></pre>
  
  <p>sample2.csv</p>
  
  <pre><code>id1|code1|name1|Lnam|mnam
  3|AAA|BBB|CCC|DDD|
  </code></pre>
  
  <p>I have compare the headers of the data frames using fuzzy logic if all header columns  2 files (sample1,sample2) average matching is 85% then i have to print both files are same.</p>
  
  <pre><code>example :
    sample1.csv vs sample2.csv  
  
  +---------+--------+-----+
  |f1_lab   |  f2_lab|score|
  +---------+--------+-----+
  | id      |   id1  |  80 |
  | code    |  code1 |  89 |
  | name    |   name1|  89 |
  | Lname   |   Lnam |  89 |
  | mname   |   mnam |  89 | 
  +---------+--------+-----+
  </code></pre>
  
  <p>my final output will be average of scores ex: <code>80+89+89+89+89/5=87.2</code> (avgage score)  . </p>
  
  <p>if average scores are above 80  the I have to print output like <code>sample1 and sample2 matched ,</code>.</p>
  
  <p>like that i have to compare all file header with other file header. I need to identify all matched files.     </p>
  
  <p>can please help me on this.</p>
  
  <p>please find below code.</p>
  
  <pre><code>from pyspark.sql import SQLContext,SparkSession
  spark = SparkSession.builder.appName("ALS").getOrCreate()
  sc = spark.sparkContext
  sqlContext = SQLContext(sc)
  df = sqlContext.read.format('com.databricks.spark.csv').options(header='True', inferschema='false').option("delimiter", "|").load('C:/Users/test/Desktop/sample1.csv')
  df1 = sqlContext.read.format('com.databricks.spark.csv').options(header='True', inferschema='false').option("delimiter", "|").load('C:/Users/test/Desktop/sample2.csv')
  df2 = sqlContext.read.format('com.databricks.spark.csv').options(header='True', inferschema='false').option("delimiter", "|").load('C:/Users/test/Desktop/sample3.csv')
  lab= [col_name for col_name in df.columns]
  lab1=[col_name1 for col_name1 in df1.columns]
  lab2=[col_name1 for col_name1 in df2.columns]
  lab1head=sc.parallelize(lab).zipWithIndex()
  a = spark.createDataFrame(lab1head,['label1', "Index"])
  lab1head=sc.parallelize(lab1).zipWithIndex()
  a1 = spark.createDataFrame(lab1head,['label2', "Index"])
  lab2head=sc.parallelize(lab1).zipWithIndex()
  a1 = spark.createDataFrame(lab2head,['label3', "Index"])
  
  from fuzzywuzzy import fuzz
  def match_name(name, list_names, min_score=0):
      max_score = -1
      # Returning empty name for no match as well
      max_name = ""
      # Iternating over all names in the other
      for name2 in list_names:
          #Finding fuzzy match score
          score = fuzz.ratio(name, name2)
          # Checking if we are above our threshold and have a better score
          if (score &gt; min_score) &amp; (score &gt; max_score):
              max_name = name2
              max_score = score
      return (max_name, max_score)
  
  dict_list = []
  for name in a.label1:
      # Use our method to find best match, we can set a threshold here
      match = match_name(name, a1.label2, 75)
      dict_ = {}
      dict_.update({"labhead" : name})
      dict_.update({"labhead1" : match[0]})
      dict_.update({"score" : match[1]})
      dict_list.append(dict_)
  </code></pre>