title: Loading shared libraries (.so) distributed by --files argument with spark
body: <p>I'm trying to work with an external native library (.so file) when running a spark job. First of all I'm submitting the file using <code>--files</code> argument.</p>
  
  <p>To load the library I'm using <code>System.load(SparkFiles.get(libname))</code> after creating the <code>SparkContext</code> (to make sure <code>SparkFiles</code> are populated).
  Problem is that the library is only loaded by the driver node, and when tasks try to access the native methods I'm getting </p>
  
  <pre><code>WARN TaskSetManager: Lost task 0.0 in stage 2.0 (TID 2, 13.0.0.206, executor 0): java.lang.UnsatisfiedLinkError
  </code></pre>
  
  <p>The only thing that worked for me was copying the <code>.so</code> file to all the workers before running the spark app, and creating a Scala object that would load the library before each task (can be optimized with <code>mapPartitions</code>).</p>
  
  <p>I tried using</p>
  
  <pre><code>--conf "spark.executor.extraLibraryPath=/local/path/to/so" \
  --conf "spark.driver.extraLibraryPath=/local/path/to/so"
  </code></pre>
  
  <p>to try to avoid that, but without success.</p>
  
  <p>Now since I'm using EMR to run spark jobs, and not a consistent cluster,
  I would like to avoid copying files to all the nodes before running the job.</p>
  
  <p>Any suggestions?</p>