title: Running an Apache Beam/Google Cloud Dataflow job from a maven-built jar
body: <p>I have a Google Cloud Dataflow job that I'm running from IntelliJ IDEA using the following command string:</p>
  
  <pre><code>compile exec:java -Dexec.mainClass=com.mygroup.mainclass "-Dexec.args=--..."
  </code></pre>
  
  <p>It runs fine from here, I want to deploy this onto a local server to be run automatically at build time. The args specify pipeline options; at any given build we need to start three different jobs using this pipeline and having to recompile three times is wasteful. So I'm using mvn package to produce a jar file in the following manner:</p>
  
  <pre><code>clean compile assembly:single
  </code></pre>
  
  <p>The problem is, when I run the pipeline via java -jar my_pipeline-1.0-SNAPSHOT-jar-with-dependencies.jar --args in the project's target directory, I'm getting an exception:</p>
  
  <pre><code>Exception in thread "main" java.lang.IllegalArgumentException:
  Unknown 'runner' specified 'DataflowRunner', supported pipeline runners [DirectRunner]
  </code></pre>
  
  <p>Posts on the Beam mailing list suggest passing -Pdataflow-runner to set the classpath at execution time, but I haven't found a way to make that work if I'm just calling the jar. I've tried specifying the profile during the compile and package steps, but that hasn't helped.</p>
  
  <p>Here's my pom.xml. It might be a little messy; I've been trying the scattershot approach to trying to make this work but nothing has stuck to the wall yet.</p>
  
  <pre><code>&lt;?xml version="1.0" encoding="UTF-8"?&gt;
  &lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"&gt;
      &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;
      &lt;groupId&gt;com.mygroup&lt;/groupId&gt;
      &lt;artifactId&gt;data_dump&lt;/artifactId&gt;
      &lt;version&gt;1.0-SNAPSHOT&lt;/version&gt;
      &lt;packaging&gt;jar&lt;/packaging&gt;
      &lt;properties&gt;
          &lt;project.build.sourceEncoding&gt;UTF-8&lt;/project.build.sourceEncoding&gt;
          &lt;maven.compiler.source&gt;1.8&lt;/maven.compiler.source&gt;
          &lt;maven.compiler.target&gt;1.8&lt;/maven.compiler.target&gt;
          &lt;slf4j.version&gt;1.7.5&lt;/slf4j.version&gt;
          &lt;junit.version&gt;4.8.2&lt;/junit.version&gt;
          &lt;hamcrest.version&gt;1.3&lt;/hamcrest.version&gt;
          &lt;mockito.version&gt;1.10.19&lt;/mockito.version&gt;
          &lt;bigquery.version&gt;v2-rev312-1.22.0&lt;/bigquery.version&gt;
          &lt;powermock.version&gt;1.6.6&lt;/powermock.version&gt;
          &lt;beam.version&gt;2.0.0&lt;/beam.version&gt;
      &lt;/properties&gt;
      &lt;build&gt;
          &lt;plugins&gt;
              &lt;plugin&gt;
                  &lt;artifactId&gt;maven-assembly-plugin&lt;/artifactId&gt;
                  &lt;configuration&gt;
                      &lt;archive&gt;
                          &lt;manifest&gt;
                              &lt;mainClass&gt;com.mygroup.mainclass&lt;/mainClass&gt;
                              &lt;addClasspath&gt;true&lt;/addClasspath&gt;
                          &lt;/manifest&gt;
                      &lt;/archive&gt;
                      &lt;descriptorRefs&gt;
                          &lt;descriptorRef&gt;jar-with-dependencies&lt;/descriptorRef&gt;
                      &lt;/descriptorRefs&gt;
                  &lt;/configuration&gt;
              &lt;/plugin&gt;
              &lt;plugin&gt;
                  &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
                  &lt;artifactId&gt;maven-compiler-plugin&lt;/artifactId&gt;
                  &lt;version&gt;3.3&lt;/version&gt;
                  &lt;configuration&gt;
                      &lt;source&gt;1.8&lt;/source&gt;
                      &lt;target&gt;1.8&lt;/target&gt;
                  &lt;/configuration&gt;
              &lt;/plugin&gt;
              &lt;plugin&gt;
                  &lt;!-- Build an executable JAR --&gt;
                  &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
                  &lt;artifactId&gt;maven-jar-plugin&lt;/artifactId&gt;
                  &lt;version&gt;3.0.2&lt;/version&gt;
                  &lt;configuration&gt;
                      &lt;archive&gt;
                          &lt;manifest&gt;
                              &lt;addClasspath&gt;true&lt;/addClasspath&gt;
                              &lt;classpathPrefix&gt;.&lt;/classpathPrefix&gt;
                              &lt;mainClass&gt;com.mygroup.mainclass&lt;/mainClass&gt;
                          &lt;/manifest&gt;
                      &lt;/archive&gt;
                  &lt;/configuration&gt;
              &lt;/plugin&gt;
              &lt;plugin&gt;
                  &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
                  &lt;artifactId&gt;maven-shade-plugin&lt;/artifactId&gt;
                  &lt;version&gt;2.4.3&lt;/version&gt;
                  &lt;executions&gt;
                      &lt;execution&gt;
                          &lt;phase&gt;package&lt;/phase&gt;
                          &lt;goals&gt;
                              &lt;goal&gt;shade&lt;/goal&gt;
                          &lt;/goals&gt;
                          &lt;configuration&gt;
                              &lt;filters&gt;
                                  &lt;filter&gt;
                                      &lt;artifact&gt;*:*&lt;/artifact&gt;
                                      &lt;excludes&gt;
                                          &lt;exclude&gt;META-INF/**&lt;/exclude&gt;
                                      &lt;/excludes&gt;
                                  &lt;/filter&gt;
                              &lt;/filters&gt;
                          &lt;/configuration&gt;
                      &lt;/execution&gt;
                  &lt;/executions&gt;
              &lt;/plugin&gt;
              &lt;plugin&gt;
                  &lt;groupId&gt;org.codehaus.mojo&lt;/groupId&gt;
                  &lt;artifactId&gt;cobertura-maven-plugin&lt;/artifactId&gt;
                  &lt;configuration&gt;
                      &lt;aggregate&gt;true&lt;/aggregate&gt;
                  &lt;/configuration&gt;
                  &lt;executions&gt;
                      &lt;execution&gt;
                          &lt;goals&gt;
                              &lt;goal&gt;clean&lt;/goal&gt;
                          &lt;/goals&gt;
                      &lt;/execution&gt;
                  &lt;/executions&gt;
              &lt;/plugin&gt;
              &lt;plugin&gt;
                  &lt;groupId&gt;org.codehaus.mojo&lt;/groupId&gt;
                  &lt;artifactId&gt;exec-maven-plugin&lt;/artifactId&gt;
                  &lt;executions&gt;
                      &lt;execution&gt;
                          &lt;goals&gt;
                              &lt;goal&gt;java&lt;/goal&gt;
                          &lt;/goals&gt;
                      &lt;/execution&gt;
                  &lt;/executions&gt;
                  &lt;configuration&gt;
                      &lt;executable&gt;java&lt;/executable&gt;
                      &lt;arguments&gt;
                      &lt;/arguments&gt;
                  &lt;/configuration&gt;
              &lt;/plugin&gt;
          &lt;/plugins&gt;
      &lt;/build&gt;
  
      &lt;profiles&gt;
          &lt;profile&gt;
              &lt;id&gt;direct-runner&lt;/id&gt;
              &lt;activation&gt;
                  &lt;activeByDefault&gt;true&lt;/activeByDefault&gt;
              &lt;/activation&gt;
              &lt;!-- Makes the DirectRunner available when running a pipeline. --&gt;
              &lt;dependencies&gt;
                  &lt;dependency&gt;
                      &lt;groupId&gt;org.apache.beam&lt;/groupId&gt;
                      &lt;artifactId&gt;beam-runners-direct-java&lt;/artifactId&gt;
                      &lt;version&gt;${beam.version}&lt;/version&gt;
                      &lt;scope&gt;runtime&lt;/scope&gt;
                  &lt;/dependency&gt;
              &lt;/dependencies&gt;
          &lt;/profile&gt;
          &lt;profile&gt;
              &lt;id&gt;dataflow-runner&lt;/id&gt;
              &lt;activation&gt;
                  &lt;activeByDefault&gt;true&lt;/activeByDefault&gt;
              &lt;/activation&gt;
              &lt;dependencies&gt;
                  &lt;dependency&gt;
                      &lt;groupId&gt;com.google.cloud.dataflow&lt;/groupId&gt;
                      &lt;artifactId&gt;google-cloud-dataflow-java-sdk-all&lt;/artifactId&gt;
                      &lt;version&gt;${beam.version}&lt;/version&gt;
                  &lt;/dependency&gt;
              &lt;/dependencies&gt;
          &lt;/profile&gt;
          &lt;profile&gt;
              &lt;id&gt;flink-runner&lt;/id&gt;
              &lt;!-- Makes the FlinkRunner available when running a pipeline. --&gt;
              &lt;dependencies&gt;
                  &lt;dependency&gt;
                      &lt;groupId&gt;org.apache.beam&lt;/groupId&gt;
                      &lt;artifactId&gt;beam-runners-flink_2.10&lt;/artifactId&gt;
                      &lt;version&gt;${beam.version}&lt;/version&gt;
                      &lt;scope&gt;runtime&lt;/scope&gt;
                  &lt;/dependency&gt;
              &lt;/dependencies&gt;
              &lt;build&gt;
                  &lt;plugins&gt;
                      &lt;plugin&gt;
                          &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
                          &lt;artifactId&gt;maven-shade-plugin&lt;/artifactId&gt;
                      &lt;/plugin&gt;
                  &lt;/plugins&gt;
              &lt;/build&gt;
          &lt;/profile&gt;
      &lt;/profiles&gt;
  
      &lt;dependencies&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.apache.beam&lt;/groupId&gt;
              &lt;artifactId&gt;beam-sdks-java-core&lt;/artifactId&gt;
              &lt;version&gt;${beam.version}&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;com.google.cloud&lt;/groupId&gt;
              &lt;artifactId&gt;google-cloud-bigquery&lt;/artifactId&gt;
              &lt;version&gt;0.18.0-beta&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;com.google.cloud&lt;/groupId&gt;
              &lt;artifactId&gt;google-cloud-storage&lt;/artifactId&gt;
              &lt;version&gt;1.0.0&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;com.google.cloud.dataflow&lt;/groupId&gt;
              &lt;artifactId&gt;google-cloud-dataflow-java-sdk-all&lt;/artifactId&gt;
              &lt;version&gt;${beam.version}&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.apache.beam&lt;/groupId&gt;
              &lt;artifactId&gt;beam-runners-direct-java&lt;/artifactId&gt;
              &lt;version&gt;${beam.version}&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.apache.beam&lt;/groupId&gt;
              &lt;artifactId&gt;beam-sdks-java-io-google-cloud-platform&lt;/artifactId&gt;
              &lt;version&gt;${beam.version}&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.apache.beam&lt;/groupId&gt;
              &lt;artifactId&gt;beam-runners-google-cloud-dataflow-java&lt;/artifactId&gt;
              &lt;version&gt;${beam.version}&lt;/version&gt;
              &lt;scope&gt;runtime&lt;/scope&gt;
          &lt;/dependency&gt;
  
          &lt;!-- slf4j API frontend binding with JUL backend --&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.slf4j&lt;/groupId&gt;
              &lt;artifactId&gt;slf4j-api&lt;/artifactId&gt;
              &lt;version&gt;${slf4j.version}&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.slf4j&lt;/groupId&gt;
              &lt;artifactId&gt;slf4j-jdk14&lt;/artifactId&gt;
              &lt;version&gt;${slf4j.version}&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;junit&lt;/groupId&gt;
              &lt;artifactId&gt;junit&lt;/artifactId&gt;
              &lt;version&gt;RELEASE&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.hamcrest&lt;/groupId&gt;
              &lt;artifactId&gt;hamcrest-library&lt;/artifactId&gt;
              &lt;version&gt;1.3&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;com.microsoft.azure&lt;/groupId&gt;
              &lt;artifactId&gt;azure-storage&lt;/artifactId&gt;
              &lt;version&gt;5.0.0&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;com.esotericsoftware.yamlbeans&lt;/groupId&gt;
              &lt;artifactId&gt;yamlbeans&lt;/artifactId&gt;
              &lt;version&gt;1.08&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;commons-io&lt;/groupId&gt;
              &lt;artifactId&gt;commons-io&lt;/artifactId&gt;
              &lt;version&gt;2.5&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.apache.commons&lt;/groupId&gt;
              &lt;artifactId&gt;commons-collections4&lt;/artifactId&gt;
              &lt;version&gt;4.1&lt;/version&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.powermock&lt;/groupId&gt;
              &lt;artifactId&gt;powermock-module-junit4&lt;/artifactId&gt;
              &lt;version&gt;${powermock.version}&lt;/version&gt;
              &lt;scope&gt;test&lt;/scope&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.powermock&lt;/groupId&gt;
              &lt;artifactId&gt;powermock-api-easymock&lt;/artifactId&gt;
              &lt;version&gt;${powermock.version}&lt;/version&gt;
              &lt;scope&gt;test&lt;/scope&gt;
          &lt;/dependency&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;org.easymock&lt;/groupId&gt;
              &lt;artifactId&gt;easymock&lt;/artifactId&gt;
              &lt;version&gt;3.4&lt;/version&gt;
          &lt;/dependency&gt;
      &lt;/dependencies&gt;
      &lt;repositories&gt;
          &lt;repository&gt;
              &lt;id&gt;apache-maven-repo&lt;/id&gt;
              &lt;name&gt;Apache Nightlies&lt;/name&gt;
              &lt;url&gt;https://repository.apache.org/content/repositories/snapshots/&lt;/url&gt;
              &lt;releases&gt;
                  &lt;enabled&gt;true&lt;/enabled&gt;
              &lt;/releases&gt;
              &lt;snapshots&gt;
                  &lt;enabled&gt;true&lt;/enabled&gt;
              &lt;/snapshots&gt;
          &lt;/repository&gt;
      &lt;/repositories&gt;
  &lt;/project&gt;
  </code></pre>