title: Finding total number of lines in hdfs distributed file using command line
body: <p>I am working on a cluster where a dataset is kept in <code>hdfs</code> in distributed manner. Here is what I have: </p>
  
  <pre><code>[hmi@bdadev-5 ~]$ hadoop fs -ls /bdatest/clm/data/
  Found 1840 items
  -rw-r--r--   3 bda supergroup          0 2015-08-11 00:32 /bdatest/clm/data/_SUCCESS
  -rw-r--r--   3 bda supergroup   34404390 2015-08-11 00:32 /bdatest/clm/data/part-00000
  -rw-r--r--   3 bda supergroup   34404062 2015-08-11 00:32 /bdatest/clm/data/part-00001
  -rw-r--r--   3 bda supergroup   34404259 2015-08-11 00:32 /bdatest/clm/data/part-00002
  ....
  ....
  </code></pre>
  
  <p>The data is of the form: </p>
  
  <pre><code>[hmi@bdadev-5 ~]$ hadoop fs -cat /bdatest/clm/data/part-00000|head
  V|485715986|1|8ca217a3d75d8236|Y|Y|Y|Y/1X||Trimode|SAMSUNG|1x/Trimode|High|Phone|N|Y|Y|Y|N|Basic|Basic|Basic|Basic|N|N|N|N|Y|N|Basic-Communicator|Y|Basic|N|Y|1X|Basic|1X|||SAM|Other|SCH-A870|SCH-A870|N|N|M2MC|
  </code></pre>
  
  <p>So, what I want to do is to count the total number of lines in the original data file <code>data</code>. My understanding is that the distributed chunks like <code>part-00000</code>, <code>part-00001</code> etc have overlaps. So just counting the number of lines in <code>part-xxxx</code> files and summing them won't work. Also the original dataset <code>data</code> is of size <code>~70GB</code>. How can I efficiently find out the total number of lines?</p>