title: Using Kafka to import data to Hadoop
body: <p>Firstly I was thinking what to use to get events into Hadoop, where they will be stored and periodically analysis would be performed on them (possibly using Ooozie to schedule periodic analysis) Kafka or Flume, and decided that Kafka is probably a better solution, since we also have a component that does event processing, so in this way, both batch and event processing components get data in the same way.</p>
  
  <p>But know I'm looking for suggestions concretely how to get data out of broker to Hadoop.</p>
  
  <p>I found <a href="https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem" rel="noreferrer">here</a> that Flume can be used in combination with Kafka </p>
  
  <ul>
  <li>Flume - Contains Kafka Source (consumer) and Sink (producer) </li>
  </ul>
  
  <p>And also found on the same page and in <a href="http://kafka.apache.org/documentation.html#introduction" rel="noreferrer">Kafka documentation</a> that there is something called <a href="https://github.com/linkedin/camus" rel="noreferrer">Camus</a></p>
  
  <ul>
  <li>Camus - LinkedIn's Kafka=>HDFS pipeline. This one is used for all data at LinkedIn, and works great.</li>
  </ul>
  
  <p>I'm interested in what would be a better (and easier, better documented solution) to do that? Also, are there any examples or tutorials how to do it?</p>
  
  <p>When should I use this variants over simpler, <em>High level consumer</em>?</p>
  
  <p>I'm opened for suggestions if there is another/better solution than this two.</p>
  
  <p>Thanks</p>