title: getting data in and out of hadoop
body: <p>I need a system to analyze large log files.  A friend directed me to hadoop the other day and it seems perfect for my needs. My question revolves around getting data into hadoop-</p>
  
  <p>Is it possible to have the nodes on my cluster stream data as they get it into HDFS?  Or would each node need to write to a local temp file and submit the temp file after it reaches a certain size?  and is it possible to append to a file in HDFS while also running queries/jobs on that same file at the same time?</p>