title: apache spark: Read large size files from a directory
body: <p>I am reading each file of a directory using <code>wholeTextFiles</code>. After that I am calling a function on each element of the rdd using <code>map</code> . The whole program uses just 50 lines of each file. The code is as below:</p>
  
  <pre><code>def processFiles(fileNameContentsPair):
    fileName= fileNameContentsPair[0]
    result = "\n\n"+fileName
    resultEr = "\n\n"+fileName
    input = StringIO.StringIO(fileNameContentsPair[1])
    reader = csv.reader(input,strict=True)
  
    try:
         i=0
         for row in reader:
           if i==50:
             break
           // do some processing and get result string
           i=i+1
    except csv.Error as e:
      resultEr = resultEr +"error occured\n\n"
      return resultEr
    return result
  
  
  
  if __name__ == "__main__":
    inputFile = sys.argv[1]
    outputFile = sys.argv[2]
    sc = SparkContext(appName = "SomeApp")
    resultRDD = sc.wholeTextFiles(inputFile).map(processFiles)
    resultRDD.saveAsTextFile(outputFile)
  </code></pre>
  
  <p>The size of each file of the directory can be very large in my case and because of this reason use of <code>wholeTextFiles</code> api will be inefficient in this case. is there any efficient way to do this ? I can think of iterating over each file of the directory one by one but that also seems to be inefficient. I am new to spark. Please let me know if there is any efficient way to do this. </p>