title: Reading data from HDFS file and sending it to a kafka producer
body: <p>I have a set of files in HDFS, these files are tweets. I need to stream these tweets through kafka producer. Later, I need to process the stream and perform some analytics over it.</p>
  
  <p>What I have done is:</p>
  
  <p>a) Write a java program to read a file line by line from HDFS.</p>
  
  <p>b) I can publish this message to a topic from a java program using kafka.</p>
  
  <p>But, this is a very slow method. And the data sizes are large.</p>
  
  <p>I looked at some mapreduce jobs from which I can feed data to the kafka producer, I am new to this and not very sure how to do that too.</p>
  
  <p>Here is what I am talking about:
  <a href="https://my-bigdata-blog.blogspot.com/2017/07/hadoop-custom-outputformat-hdfs-send-to-kafka.html" rel="nofollow noreferrer">https://my-bigdata-blog.blogspot.com/2017/07/hadoop-custom-outputformat-hdfs-send-to-kafka.html</a></p>