title: Retrieving data from twitter using flume and storing to hdfs in JSON FORMAT
body: <p>I am trying to retrieve data from twitter using flume and storing to hdfs in JSON FORMAT.And the data is loading to HDFS.BUT NOT IN JSON FORMAT.</p>
  
  <p>I am attaching few lines from the HDFS file which is stored from twitter :</p>
  
  <pre><code>Objavro.schema\E4
  {"type":"record","name":"Doc","doc":"adoc","fields":[{"name":"id","type":"string"},{"name":"user_friends_count","type":["int","null"]},{"name":"user_location","type":["string","null"]},{"name":"user_description","type":["string","null"]},{"name":"user_statuses_count","type":["int","null"]},{"name":"user_followers_count","type":["int","null"]},{"name":"user_name","type":["string","null"]},{"name":"user_screen_name","type":["string","null"]},{"name":"created_at","type":["string","null"]},{"name":"text","type":["string","null"]},{"name":"retweet_count","type":["long","null"]},{"name":"retweeted","type":["boolean","null"]},{"name":"in_reply_to_user_id","type":["long","null"]},{"name":"source","type":["string","null"]},{"name":"in_reply_to_status_id","type":["long","null"]},{"name":"media_url_https","type":["string","null"]},{"name":"expanded_url","type":["string","null"]}]}\00\E0D\C9H\B8$\DCb,C\8A5y\D1n\CE$733267766577356800\00\96\00Zumaran \00\C6C.A.B//C.A.H
  Wsp:351 220-1251
  Fb:Ramiro Pedernera✌
  Insta:Ramiropedernera
  Snapp:ramipedernera12\00\B2\9E\00\B2(\00(DIVI^Lista RAMIRO P.\00RamiPedernera12\00(2016-05-19T17:37:13Z\00tGaray culiadaso me metió una patada en la frente \00\00\00\00\00\00\A8&lt;a href="http://twitter.com/download/android" rel="nofollow"&gt;Twitter for Android&lt;/a&gt;\00\E0D\C9H\B8$\DCb,C\8A5y\D1n
  Objavro.schema\E4
  </code></pre>
  
  <p>Since this is not in JSON format its not possible to process it by creating table in HIVE and loading this data.
  So Please help me to load the twitter data in JSON format to HADOOP HDFS</p>
  
  <p>This is the command I used :</p>
  
  <pre><code>bin/flume-ng agent --conf ./conf/ -f conf/twitter.conf -Dflume.root.logger=DEBUG,console -n TwitterAgent
  </code></pre>
  
  <p>And the twitter.conf is attached:</p>
  
  <pre><code>TwitterAgent.sources = Twitter
  TwitterAgent.channels = MemChannel
  TwitterAgent.sinks = HDFS
  TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
  TwitterAgent.sources.Twitter.channels = MemChannel
  TwitterAgent.sources.Twitter.consumerKey =********
  TwitterAgent.sources.Twitter.consumerSecret =*************
  TwitterAgent.sources.Twitter.accessToken =****************
  TwitterAgent.sources.Twitter.accessTokenSecret =*****************
  TwitterAgent.sources.Twitter.keywords = hadoop, big data, analytics, bigdata, cloudera, data science, data scientiest, business intelligence, mapreduce, data warehouse, data warehousing, mahout, hbase, nosql, newsql, businessintelligence, cloudcomputing
  TwitterAgent.sinks.HDFS.channel = MemChannel
  TwitterAgent.sinks.HDFS.type = hdfs
  TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:54310/user/hduser_/twitter-cool
  TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
  TwitterAgent.sinks.HDFS.hdfs.writeFormat = json
  TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
  TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
  TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
  TwitterAgent.channels.MemChannel.type = memory
  TwitterAgent.channels.MemChannel.capacity = 10000
  TwitterAgent.channels.MemChannel.transactionCapacity = 100
  TwitterAgent.sources.Twitter.handler = org.apache.flume.source.http.JSONHandler
  </code></pre>