Hadoop: Sending Files or File paths to a map reduce job
Question: supposed I had N files to process using hadoop map-reduce, let's assume they
are large, well beyond the block size and there are only a few hundred of
them. Now I would like to process each of these files, let's assume the word
counting example.

My question is: What is the difference between creating a map-reduce job whose
input is a text file with the paths to each of the files as opposed to sending
each of the files directly to the map function i.e. concatenating all the
files and pushing **them into different mappers [EDIT]**.

Are these both valid approaches? Are there any drawbacks to them?

**Thanks for the prompt answers I've included a detailed description of my
problem since my abstraction may have missed a few important topics:**

I have N small files on Hadoop HDFS in my application and I just need to
process each file. So I am using a map function to apply a python script to
each file (actually image [I've already looked at all the hadoop image
processing links out there]), I am aware of the small file problem and the
typical recommendation is to group the smaller files so we avoid the overhead
of moving files around (the basic recommendation using sequence files or
creating your own data structures as in the case of the HIPI).

_**This makes me wonder can't we just tell each mapper to look for files that
are local to him and operate on those?_**

I haven't found a solution to that issue which is why I was looking at either
sending a path to the files to each mapper or the file it self.

Creating a list of path names for each of the collection of images seems to be
ok, but as mentioned in the comments I loose the data locality property.

Now when I looked at the hadoop streaming interface it mentions that the
different pieces communicate based on stdin and stdout typically used for text
files. That's where I get confused, if I am just sending a list of path names
this shouldn't be an issue since each mapper would just try to find the
collection of images it is assigned. But when I look at the word count example
the input is the file which then gets split up across the mappers and **so
that's when I was confused as to if I should concatenate images into groups
and then send these concatenated groups just like the text document to the
different mappers or if I should instead concatenate the images leave them in
hadoop HDFS and then just pass the path to them to the mapper**... I hope this
makes sense... maybe I'm completely off here...

Thanks again!


Answer: Both are valid. But latter would incur extra overhead and performance will go
down because you are talking about concatenating all the files into one and
feeding it to just 1 mapper. And by doing that you would go against one of the
most basic principles of Hadoop, `parallelism`. Parallelism is what makes
Hadoop so efficient.

FYI, if you really need to do that you have to set `isSplittable` to false in
your `InputFormat` class, otherwise the framework will split the file(based on
your InputFormat).

And as far as input path is considered, you just need to give the path of the
input directory. Each file inside this directory will be processed without
human intervention.

HTH

* * *

In response to your edit :

I think you have misunderstood this a bit. You don't have to worry about
localization. Hadoop takes care of that. You just have to run your job and the
data will be processed on the node where it is present. Size of the file has
nothing to with it. You don't have to tell anything to mappers. Process goes
like this :

You submit your job to JT. JT directs the TT running on the node which has the
block of data required by the job to start start the mappers. If the slots are
occupied by some other process, then same thing takes place on some other node
having the data block.

