title: Running external library with Cloud Dataflow
body: <p>I'm trying to run some external shared library functions with Cloud Dataflow similar to described here: <a href="https://cloud.google.com/blog/big-data/2017/07/running-external-libraries-with-cloud-dataflow-for-grid-computing-workloads" rel="nofollow noreferrer">Running external libraries with Cloud Dataflow for grid-computing workloads</a>.</p>
  
  <p>I have a <em>couple</em> of questions according to the approach.</p>
  
  <p>There is the following passage in the article <a href="https://cloud.google.com/blog/big-data/2017/07/running-external-libraries-with-cloud-dataflow-for-grid-computing-workloads" rel="nofollow noreferrer">mentioned earlier</a>:</p>
  
  <blockquote>
    <p>In the case of making a call to an external library, you need to do this step manually for that library. The approach is to:</p>
    
    <ul>
    <li>Store the code (along with versioning information) in Cloud Storage, this removes any concerns about throughput if running 10,000s of cores in the flow.</li>
    <li>In the <strong><code>@beginBundle</code></strong> <em>[sic]</em> method, create a synchronized block to check if the file is available on the local resource. If not, use the Cloud Storage client library to pull the file across.</li>
    </ul>
  </blockquote>
  
  <p>However, with my Java package, I simply put the library <code>.so</code> file into the <code>src/main/resource/linux-x86-64</code> directory and call the library functions the following way <em>(stripped to a bare minimum for brevity)</em>:</p>
  
  <pre><code>import com.sun.jna.Library;
  import com.sun.jna.Native;
  import org.apache.beam.sdk.transforms.DoFn;
  import org.apache.beam.sdk.values.KV;
  
  public class HostLookupPipeline {
  
    public interface LookupLibrary extends Library {
      String Lookup(String domain);
    }
  
    static class LookupFn extends DoFn&lt;String, KV&lt;String, String&gt;&gt; {
      private static LookupLibrary lookup;
  
      @StartBundle
      public void startBundle() {
        // src/main/resource/linux-x86-64/liblookup.so
        lookup = Native.loadLibrary("lookup", LookupLibrary.class);
      }
  
      @ProcessElement
      public void processElement(ProcessContext c) {
        String domain = c.element();
        String results = lookup.Lookup(domain);
        if (results != null) {
          c.output(KV.of(domain, results));
        }
      }
    }
  }
  </code></pre>
  
  <p><strong>Is such approach considered acceptable or extracting <code>.so</code> file from JAR performs poorly compared to downloading from GCS? If not, where should I put the file after downloading to make it accessible by the Cloud Dataflow worker?</strong></p>
  
  <p>I've noticed that the transformation calling the external library function works rather slow — <em>about 90 elements/s</em> — utilizing 15 Cloud Dataflow workers (autoscaling, default max workers). If my rough calculations are correct, it should be twice as fast. I suppose that's because I call the external library function for every element.</p>
  
  <p><strong>Are there any best practices to improve external libraries performance when running with Java?</strong></p>