title: How to skip lines while reading a CSV file as a dataFrame using PySpark?
body: <p>I have a CSV file that is structured this way:</p>
  
  <pre><code>Header
  Blank Row
  "Col1","Col2"
  "1,200","1,456"
  "2,000","3,450"
  </code></pre>
  
  <p>I have two problems in reading this file. </p>
  
  <ol>
  <li>I want to Ignore the Header and Ignore the blank row</li>
  <li>The commas within the value is not a separator</li>
  </ol>
  
  <p>Here is what I tried: </p>
  
  <pre><code>df = sc.textFile("myFile.csv")\
                .map(lambda line: line.split(","))\ #Split By comma
                .filter(lambda line: len(line) == 2).collect() #This helped me ignore the first two rows
  </code></pre>
  
  <p>However, This did not work, because the commas within the value was being read as a separator and the <code>len(line)</code> was returning 4 instead of 2. </p>
  
  <p>I tried an alternate approach: </p>
  
  <pre><code>data = sc.textFile("myFile.csv")
  headers = data.take(2) #First two rows to be skipped
  </code></pre>
  
  <p>The idea was to then use filter and not read the headers. But, when I tried to print the headers, I got encoded values. </p>
  
  <pre><code>[\x00A\x00Y\x00 \x00J\x00u\x00l\x00y\x00 \x002\x000\x001\x006\x00]
  </code></pre>
  
  <p>What is the correct way to read a CSV file and skip the first two rows? </p>