title: Retrieving external .proto files for protoc with protobuf-maven-plugin
body: <p>Context: My team is working on making our suite of Java services containerized and dynamically scalable. To accomplish this, our plan is to use Envoy backed by etcd, with a custom-build <a href="https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/eds.proto.html#envoy-api-file-envoy-api-v2-eds-proto" rel="nofollow noreferrer" title="Endpoint Discovery Service">Endpoint Discovery Service</a> as described in the <a href="https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/dynamic_configuration.html#arch-overview-dynamic-config-sds" rel="nofollow noreferrer" title="Envoy documentation">Envoy documentation</a>, using the <a href="https://www.envoyproxy.io/docs/envoy/latest/configuration/overview/v2_overview.html" rel="nofollow noreferrer" title="v2 Envoy gRPC API">v2 gRPC-based API</a>. We'll then generate Docker images for each service and deploy/manage them with Kubernetes.  </p>
  
  <p>We use Maven as a build system. I'm pretty well versed in Maven, but this is my first time with gRPC or protocol buffers.</p>
  
  <p>I've created a stub for my service using Spring Boot, with Jetty offering some REST and JMX endpoints for management. Before introducing the protobuf stuff, the stub built and ran just fine.</p>
  
  <p>I've downloaded the <a href="https://github.com/envoyproxy/data-plane-api/" rel="nofollow noreferrer" title="Envoy data-plane-api">Envoy data-plane-api</a> and checked the API definition files (**/*.proto) into my project under src/main/proto, keeping the directory structure from the download (e.g. src/main/proto/envoy/api/v2/eds.proto). (Side question: do I need the BUILD files?)</p>
  
  <p>In the end, I'd like a standalone Maven build that can read these files and generate Java classes. The build needs to work on Windows and OS X boxes so that it works for developers, and on Linux boxes so that it works in our CI (Bamboo). It should require nothing more than a JDK, a Maven install, and a Maven repo. (We have an <a href="https://jfrog.com/artifactory/" rel="nofollow noreferrer">Artifactory</a> instance where I can upload artifacts not otherwise available online, if necessary.)</p>
  
  <p>What I have so far seems like it will accomplish my portability goals:</p>
  
  <p>pom.xml:</p>
  
  <pre><code>&lt;?xml version="1.0" encoding="UTF-8"?&gt;
  &lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"&gt;
  
      &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;
      &lt;artifactId&gt;discovery-service&lt;/artifactId&gt;
      &lt;packaging&gt;jar&lt;/packaging&gt;
      &lt;name&gt;Discovery Service&lt;/name&gt;
      &lt;version&gt;1.0.0&lt;/version&gt;
  
      &lt;build&gt;
          &lt;plugins&gt;
              &lt;plugin&gt;
                  &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
                  &lt;artifactId&gt;maven-compiler-plugin&lt;/artifactId&gt;
                  &lt;version&gt;3.7.0&lt;/version&gt;
                  &lt;configuration&gt;
                      &lt;verbose&gt;true&lt;/verbose&gt;
                      &lt;fork&gt;true&lt;/fork&gt;
                      &lt;source&gt;1.8&lt;/source&gt;
                      &lt;target&gt;1.8&lt;/target&gt;
                  &lt;/configuration&gt;
              &lt;/plugin&gt;
              &lt;plugin&gt;
                  &lt;groupId&gt;org.xolstice.maven.plugins&lt;/groupId&gt;
                  &lt;artifactId&gt;protobuf-maven-plugin&lt;/artifactId&gt;
                  &lt;version&gt;0.5.1&lt;/version&gt;
                  &lt;configuration&gt;
                      &lt;protocArtifact&gt;com.google.protobuf:protoc:3.5.1-1:exe:${os.detected.classifier}&lt;/protocArtifact&gt;
                      &lt;checkStaleness&gt;true&lt;/checkStaleness&gt;
                  &lt;/configuration&gt;
                  &lt;executions&gt;
                      &lt;execution&gt;
                          &lt;goals&gt;
                              &lt;goal&gt;compile&lt;/goal&gt;
                              &lt;goal&gt;test-compile&lt;/goal&gt;
                          &lt;/goals&gt;
                      &lt;/execution&gt;
                  &lt;/executions&gt;
              &lt;/plugin&gt;
          &lt;/plugins&gt;
          &lt;extensions&gt;
              &lt;!-- provides os.detected.classifier (i.e. linux-x86_64, osx-x86_64) property --&gt;
              &lt;extension&gt;
                  &lt;groupId&gt;kr.motd.maven&lt;/groupId&gt;
                  &lt;artifactId&gt;os-maven-plugin&lt;/artifactId&gt;
                  &lt;version&gt;1.4.1.Final&lt;/version&gt;
              &lt;/extension&gt;
          &lt;/extensions&gt;
      &lt;/build&gt;
  
      &lt;dependencies&gt;
          &lt;dependency&gt;
              &lt;groupId&gt;io.hydrosphere&lt;/groupId&gt;
              &lt;artifactId&gt;envoy-data-plane-api_2.11&lt;/artifactId&gt;
              &lt;version&gt;v1.5.0_1&lt;/version&gt;
          &lt;/dependency&gt;
  
          &lt;dependency&gt;
              &lt;groupId&gt;com.google.protobuf&lt;/groupId&gt;
              &lt;artifactId&gt;protobuf-java&lt;/artifactId&gt;
              &lt;version&gt;3.5.1&lt;/version&gt;
          &lt;/dependency&gt;
  
          &lt;dependency&gt;
              &lt;groupId&gt;io.grpc&lt;/groupId&gt;
              &lt;artifactId&gt;grpc-core&lt;/artifactId&gt;
              &lt;version&gt;1.9.1&lt;/version&gt;
          &lt;/dependency&gt;
  
          &lt;dependency&gt;
              &lt;groupId&gt;io.grpc&lt;/groupId&gt;
              &lt;artifactId&gt;grpc-protobuf&lt;/artifactId&gt;
              &lt;version&gt;1.9.1&lt;/version&gt;
          &lt;/dependency&gt;
  
          &lt;!--Unrelated deps (Spring Boot, Jetty, logging, etc) omitted for brevity --&gt;
  
      &lt;/dependencies&gt;
  &lt;/project&gt;
  </code></pre>
  
  <p>When I build this project using <code>mvn compile</code>, Maven correctly downloads <code>protoc</code> and invokes it, but protoc errors because it can't find some external dependencies:</p>
  
  <pre><code>[INFO] Scanning for projects...
  [INFO] ------------------------------------------------------------------------
  [INFO] Detecting the operating system and CPU architecture
  [INFO] ------------------------------------------------------------------------
  [INFO] os.detected.name: osx
  [INFO] os.detected.arch: x86_64
  [INFO] os.detected.classifier: osx-x86_64
  [INFO] 
  [INFO] ------------------------------------------------------------------------
  [INFO] Building Discovery Service 1.0.0
  [INFO] ------------------------------------------------------------------------
  Downloading: http://devutl1.sircon.com:8081/artifactory/libs-snapshot/org/glassfish/javax.el/maven-metadata.xml
  Downloading: http://devutl1.sircon.com:8081/artifactory/libs-release/org/glassfish/javax.el/maven-metadata.xml
  Downloaded: http://devutl1.sircon.com:8081/artifactory/libs-release/org/glassfish/javax.el/maven-metadata.xml (882 B at 547 B/s)
  Downloaded: http://devutl1.sircon.com:8081/artifactory/libs-snapshot/org/glassfish/javax.el/maven-metadata.xml (882 B at 547 B/s)
  [INFO] 
  [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (default-cli) @ discovery-service ---
  [INFO] 
  [INFO] --- protobuf-maven-plugin:0.5.1:compile (default) @ discovery-service ---
  [INFO] Compiling 56 proto file(s) to /Users/jrobb/Projects/vertabrae/trunk/scaling/discovery-service/target/generated-sources/protobuf/java
  [ERROR] PROTOC FAILED: validate/validate.proto: File not found.
  gogoproto/gogo.proto: File not found.
  envoy/api/v2/core/address.proto: Import "validate/validate.proto" was not found or had errors.
  envoy/api/v2/core/address.proto: Import "gogoproto/gogo.proto" was not found or had errors.
  envoy/config/metrics/v2/stats.proto: Import "envoy/api/v2/core/address.proto" was not found or had errors.
  envoy/config/metrics/v2/stats.proto: Import "validate/validate.proto" was not found or had errors.
  envoy/config/metrics/v2/stats.proto:148:5: "envoy.api.v2.core.Address" is not defined.
  envoy/config/metrics/v2/stats.proto:167:5: "envoy.api.v2.core.Address" is not defined.
  </code></pre>
  
  <p>The output continues, with the same set of errors for each <code>.proto</code> file.</p>
  
  <p>It seems that the crux of the issue is that I don't have validate/validate.proto or gogoproto/gogo.proto, which are imported at the top of (nearly?) every .proto file:</p>
  
  <pre><code>import "google/protobuf/wrappers.proto";
  
  import "validate/validate.proto";
  import "gogoproto/gogo.proto";
  </code></pre>
  
  <p>It seems to be finding <code>wrappers.proto</code>, which I think is coming from my compile-time Maven dependency on <code>protobuf-java</code>.</p>
  
  <p>I think <code>gogoproto/gogo.proto</code> might be looking for this: <a href="https://github.com/gogo/protobuf/blob/master/gogoproto/gogo.proto" rel="nofollow noreferrer">https://github.com/gogo/protobuf/blob/master/gogoproto/gogo.proto</a></p>
  
  <p>I'm totally stumped about where <code>validate/validate.proto</code> is supposed to come from. I've seen some evidence that it's specific to Envoy, but I can't find it.</p>
  
  <p>I've spent the last few hours searching, and I'm coming up empty on anyone who's done this before. Where do I get those files? If I can't get them from Maven Central, I'll build them myself and upload to Artifactory.</p>
  
  <p>My eyes are glazing over looking at documentation intended for people using other tech stacks, and I'm having trouble translating for my needs. Any and all help will be much appreciated, and I apologize if this is a hugely n00bish question. :)</p>