title: SparkJob on GCP dataproc failing with error - java.lang.NoSuchMethodError: io.netty.buffer.PooledByteBufAllocator.<init>(ZIIIIIIZ)V
body: <p>I'm running a spark job on GCP dataproc using below command,</p>
  <pre><code>gcloud dataproc workflow-templates instantiate-from-file --file=job_config.yaml  --region us-east1
  </code></pre>
  <p>Below is my job_config.yaml</p>
  <pre><code>jobs:
   - sparkJob:
  args:
  - filepath
  mainJarFileUri: gs://&lt;host&gt;/my-project-1.0.0-SNAPSHOT.jar
  
  stepId: sparkjob
  placement:
    managedCluster:
  clusterName: mysparkcluster
  config:
    configBucket: my-dataproc-spark
    gceClusterConfig:
      zoneUri: 'us-east1-c'
      subnetworkUri: '&lt;uri&gt;'      
    masterConfig:
      diskConfig:
        bootDiskSizeGb: 30
        bootDiskType: 'pd-ssd'
      machineTypeUri: 'e2-standard-4'
    workerConfig:
      diskConfig:
        bootDiskSizeGb: 30
        bootDiskType: 'pd-ssd'
      machineTypeUri: 'e2-standard-4'
      numInstances: 2
    secondaryWorkerConfig:
      diskConfig:
        bootDiskType: 'pd-ssd'
      numInstances: 2
    softwareConfig:
      imageVersion: '2.0.44-debian10'
      properties:
        dataproc:dataproc.logging.stackdriver.job.driver.enable: 'true'
        dataproc:dataproc.logging.stackdriver.job.yarn.container.enable: 'true'
        dataproc:dataproc.monitoring.stackdriver.enable: 'true'
        dataproc:dataproc.scheduler.driver-size-mb: '256'
        dataproc:ranger.cloud-sql.use-private-ip: 'true'
        spark:spark.dynamicAllocation.enabled: 'false'
        spark:spark.shuffle.service.enabled: 'false'
        spark:spark.executor.instances: '30'
        spark:spark.executor.memory: '9g'
        spark:spark.executors.cores: '3'
  </code></pre>
  <p>Below is my pom.xml:</p>
  <pre><code>&lt;project xmlns=&quot;http://maven.apache.org/POM/4.0.0&quot; xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot;
       xsi:schemaLocation=&quot;http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd&quot;&gt;
  &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;
  &lt;groupId&gt;com.sai.spark&lt;/groupId&gt;
  &lt;artifactId&gt;spark-project&lt;/artifactId&gt;
  &lt;version&gt;0.0.1-SNAPSHOT&lt;/version&gt;
  &lt;packaging&gt;jar&lt;/packaging&gt;
  
  &lt;dependencies&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;com.google.cloud.spark&lt;/groupId&gt;
          &lt;artifactId&gt;spark-bigquery_2.12&lt;/artifactId&gt;
          &lt;version&gt;0.22.0&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
          &lt;artifactId&gt;spark-core_2.12&lt;/artifactId&gt;
          &lt;version&gt;3.1.2&lt;/version&gt;
          &lt;scope&gt;compile&lt;/scope&gt;
          &lt;exclusions&gt;
              &lt;exclusion&gt;
                  &lt;groupId&gt;com.google.code.gson&lt;/groupId&gt;
                  &lt;artifactId&gt;gson&lt;/artifactId&gt;
              &lt;/exclusion&gt;
          &lt;/exclusions&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;com.google.protobuf&lt;/groupId&gt;
          &lt;artifactId&gt;protobuf-java&lt;/artifactId&gt;
          &lt;version&gt;3.11.4&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.scala-lang&lt;/groupId&gt;
          &lt;artifactId&gt;scala-library&lt;/artifactId&gt;
          &lt;version&gt;2.12.14&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.httpcomponents&lt;/groupId&gt;
          &lt;artifactId&gt;httpclient&lt;/artifactId&gt;
          &lt;version&gt;4.5.4&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
          &lt;artifactId&gt;spark-sql_2.12&lt;/artifactId&gt;
          &lt;version&gt;3.1.2&lt;/version&gt;
          &lt;scope&gt;compile&lt;/scope&gt;
      &lt;/dependency&gt;
  
      &lt;dependency&gt;
          &lt;groupId&gt;org.elasticsearch&lt;/groupId&gt;
          &lt;artifactId&gt;elasticsearch&lt;/artifactId&gt;
          &lt;version&gt;7.5.2&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;com.google.cloud&lt;/groupId&gt;
          &lt;artifactId&gt;google-cloud-secretmanager&lt;/artifactId&gt;
          &lt;version&gt;0.3.0&lt;/version&gt;
          &lt;exclusions&gt;
              &lt;exclusion&gt;
                  &lt;groupId&gt;io.grpc&lt;/groupId&gt;
                  &lt;artifactId&gt;grpc-grpclb&lt;/artifactId&gt;
              &lt;/exclusion&gt;
          &lt;/exclusions&gt;
      &lt;/dependency&gt;
      &lt;!-- https://mvnrepository.com/artifact/com.typesafe/config --&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;com.typesafe&lt;/groupId&gt;
          &lt;artifactId&gt;config&lt;/artifactId&gt;
          &lt;version&gt;1.4.0&lt;/version&gt;
      &lt;/dependency&gt;
  
  &lt;/dependencies&gt;
  &lt;build&gt;
      &lt;sourceDirectory&gt;src/main/scala&lt;/sourceDirectory&gt;
      &lt;plugins&gt;
        
          &lt;plugin&gt;
              &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
              &lt;artifactId&gt;maven-shade-plugin&lt;/artifactId&gt;
              &lt;version&gt;3.2.3&lt;/version&gt;
              &lt;executions&gt;
                  &lt;execution&gt;
                      &lt;phase&gt;package&lt;/phase&gt;
                      &lt;goals&gt;
                          &lt;goal&gt;shade&lt;/goal&gt;
                      &lt;/goals&gt;
                      &lt;configuration&gt;
                          &lt;createDependencyReducedPom&gt;false&lt;/createDependencyReducedPom&gt;
                          &lt;artifactSet&gt;
                              &lt;includes&gt;
                                  &lt;!-- dependencies to be packed in the fat jar --&gt;
                                  &lt;include&gt;*&lt;/include&gt;
                              &lt;/includes&gt;
                          &lt;/artifactSet&gt;
                          &lt;transformers&gt;
                              &lt;transformer
                                      implementation=&quot;org.apache.maven.plugins.shade.resource.AppendingTransformer&quot;&gt;
                                  &lt;resource&gt;reference.conf&lt;/resource&gt;
                              &lt;/transformer&gt;
                              &lt;transformer
                                      implementation=&quot;org.apache.maven.plugins.shade.resource.ManifestResourceTransformer&quot;&gt;
                                  &lt;manifestEntries&gt;
                                      &lt;Main-Class&gt;com.sai.spark.Test&lt;/Main-Class&gt;
                                  &lt;/manifestEntries&gt;
                              &lt;/transformer&gt;
                              &lt;transformer
                                      implementation=&quot;org.apache.maven.plugins.shade.resource.ServicesResourceTransformer&quot;/&gt;
                          &lt;/transformers&gt;
                          &lt;filters&gt;
                              &lt;filter&gt;
                                  &lt;artifact&gt;*:*&lt;/artifact&gt;
                                  &lt;excludes&gt;
                                      &lt;exclude&gt;META-INF/maven/**&lt;/exclude&gt;
                                      &lt;exclude&gt;META-INF/*.SF&lt;/exclude&gt;
                                      &lt;exclude&gt;META-INF/*.DSA&lt;/exclude&gt;
                                      &lt;exclude&gt;META-INF/*.RSA&lt;/exclude&gt;
                                  &lt;/excludes&gt;
                              &lt;/filter&gt;
                          &lt;/filters&gt;
                          &lt;relocations&gt;
                              &lt;relocation&gt;
                                  &lt;pattern&gt;com&lt;/pattern&gt;
                                  &lt;shadedPattern&gt;repackaged.com&lt;/shadedPattern&gt;
                                  &lt;includes&gt;
                                      &lt;include&gt;com.google.protobuf.**&lt;/include&gt;
                                      &lt;include&gt;com.google.common.**&lt;/include&gt;
                                      &lt;include&gt;com.google.gson.**&lt;/include&gt;
                                      &lt;include&gt;com.google.api.**&lt;/include&gt;
                                  &lt;/includes&gt;
                              &lt;/relocation&gt;
                             
                              &lt;relocation&gt;
                                  &lt;pattern&gt;org&lt;/pattern&gt;
                                  &lt;shadedPattern&gt;repackaged.org&lt;/shadedPattern&gt;
                                  &lt;includes&gt;
                                      &lt;include&gt;org.apache.commons.**&lt;/include&gt;
                                  &lt;/includes&gt;
                              &lt;/relocation&gt;
                          &lt;/relocations&gt;
                      &lt;/configuration&gt;
                  &lt;/execution&gt;
              &lt;/executions&gt;
          &lt;/plugin&gt;
      &lt;/plugins&gt;
  &lt;/build&gt;
  </code></pre>
  
  <p>StackTrace:</p>
  <pre><code>22/07/01 16:58:00 ERROR io.grpc.internal.ManagedChannelImpl: [Channel&lt;1&gt;: (secretmanager.googleapis.com:443)] Uncaught exception in the SynchronizationContext. Panic!
  java.lang.NoSuchMethodError: io.netty.buffer.PooledByteBufAllocator.&lt;init&gt;(ZIIIIIIZ)V
  at io.grpc.netty.Utils.createByteBufAllocator(Utils.java:172)
  at io.grpc.netty.Utils.access$000(Utils.java:71)
  at io.grpc.netty.Utils$ByteBufAllocatorPreferDirectHolder.&lt;clinit&gt;(Utils.java:93)
  at io.grpc.netty.Utils.getByteBufAllocator(Utils.java:140)
  at io.grpc.netty.NettyClientTransport.start(NettyClientTransport.java:231)
  at io.grpc.internal.ForwardingConnectionClientTransport.start(ForwardingConnectionClientTransport.java:33)
  at io.grpc.internal.ForwardingConnectionClientTransport.start(ForwardingConnectionClientTransport.java:33)
  at io.grpc.internal.InternalSubchannel.startNewTransport(InternalSubchannel.java:258)
  at io.grpc.internal.InternalSubchannel.access$400(InternalSubchannel.java:65)
  at io.grpc.internal.InternalSubchannel$2.run(InternalSubchannel.java:200)
  at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:95)
  at io.grpc.SynchronizationContext.execute(SynchronizationContext.java:127)
  at io.grpc.internal.ManagedChannelImpl$NameResolverListener.onResult(ManagedChannelImpl.java:1852)
  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:333)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:750)
  Exception in thread &quot;main&quot; repackaged.com.google.api.gax.rpc.InternalException: io.grpc.StatusRuntimeException: INTERNAL: Panic! This is a bug!
  at repackaged.com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:67)
  at repackaged.com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
  at repackaged.com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
  at repackaged.com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
  at repackaged.com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
  at repackaged.com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1074)
  at repackaged.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
  at repackaged.com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)
  at repackaged.com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)
  at repackaged.com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)
  at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:522)
  at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:497)
  at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463)
  at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:427)
  at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:460)
  at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:553)
  at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:68)
  at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:739)
  at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:718)
  at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
  at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:750)
  </code></pre>
  <p>I have tried solutions specified in below stackoverflow questions, but issue is not getting resolved.</p>
  <ul>
  <li><a href="https://stackoverflow.com/questions/47582740/netty-version-conflict-with-spark-elasticsearch-transport">Netty Version Conflict with Spark + Elasticsearch Transport</a></li>
  <li><a href="https://stackoverflow.com/questions/50055656/java-sparkcontext-error-java-lang-nosuchmethoderror-io-netty-buffer-pooledbyte">Java SparkContext error: java.lang.NoSuchMethodError: io.netty.buffer.PooledByteBufAllocator</a></li>
  <li><a href="https://stackoverflow.com/questions/49137397/spark-2-3-0-netty-version-issue-nosuchmethod-io-netty-buffer-pooledbytebufalloc">Spark 2.3.0 netty version issue: NoSuchMethod io.netty.buffer.PooledByteBufAllocator.metric()</a></li>
  </ul>