title: Hadoop - submit a job with lots of dependencies (jar files)
body: <p>I want to write some sort of "bootstrap" class, which will watch MQ for incoming messages and submit map/reduce jobs to Hadoop. These jobs use some external libraries heavily. For the moment I have the implementation of these jobs, packaged as ZIP file with bin,lib and log folders (I'm using maven-assembly-plugin to tie things together).</p>
  
  <p>Now I want to provide small wrappers for Mapper and Reducer, which will use parts of the existing application.</p>
  
  <p>As far as I learned, when a job is submitted, Hadoop tries to find out JAR file, which has the  mapper/reducer classes, and copy this jar over network to data node, which will be used to process the data. But it's not clear how do I tell Hadoop to copy all dependencies?</p>
  
  <p>I could use maven-shade-plugin to create an uber-jar with the job and dependencies, And another jar for bootstrap (which jar would be executed with hadoop shell-script).</p>
  
  <p>Please advice.</p>