title: Remote Spark standalone executor error
body: <p>I am running a spark (2.0.1) standalone cluster on a remote server (Microsoft azure). I am able to connect my spark app to this cluster, however the tasks are getting stuck without any execution(with the following warning: 
  <code>WARN org.apache.spark.scheduler.TaskSchedulerImpl - Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources</code>) </p>
  
  <p>What I have tried:</p>
  
  <ol>
  <li><p>I have ensured that the memory, cpu requirements for my app do not exceed the server config.</p></li>
  <li><p>Have supplied these variables to my <code>spark-env.sh</code> : <code>SPARK_PUBLIC_DNS ,SPARK_DRIVER_HOST, SPARK_LOCAL_IP, SPARK_MASTER_HOST</code></p></li>
  <li>Can see the master / worker / application webui on the browser.</li>
  <li>Have all the ports open on the remote server (for my IP and the vpn).</li>
  <li>Disabled <code>ufw</code>.</li>
  </ol>
  
  <p>As far as I can tell, my workers are not able to relay back to the master. Executors are timing out after 120s with the following stderr: </p>
  
  <pre><code>Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  16/11/19 18:15:09 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 17261@sparkmasternew
  16/11/19 18:15:09 INFO SignalUtils: Registered signal handler for TERM
  16/11/19 18:15:09 INFO SignalUtils: Registered signal handler for HUP
  16/11/19 18:15:09 INFO SignalUtils: Registered signal handler for INT
  16/11/19 18:15:10 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  16/11/19 18:15:10 INFO SecurityManager: Changing view acls to: ubuntu,user1
  16/11/19 18:15:10 INFO SecurityManager: Changing modify acls to: ubuntu,user1
  16/11/19 18:15:10 INFO SecurityManager: Changing view acls groups to: 
  16/11/19 18:15:10 INFO SecurityManager: Changing modify acls groups to: 
  16/11/19 18:15:10 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(ubuntu, user1); groups with view permissions: Set(); users  with modify permissions: Set(ubuntu, user1); groups with modify permissions: Set()
  java.lang.IllegalArgumentException: requirement failed: TransportClient has not yet been set.
      at scala.Predef$.require(Predef.scala:224)
      at org.apache.spark.rpc.netty.RpcOutboxMessage.onTimeout(Outbox.scala:70)
      at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$ask$1.applyOrElse(NettyRpcEnv.scala:232)
      at org.apache.spark.rpc.netty.NettyRpcEnv$$anonfun$ask$1.applyOrElse(NettyRpcEnv.scala:231)
      at scala.concurrent.Future$$anonfun$onFailure$1.apply(Future.scala:138)
      at scala.concurrent.Future$$anonfun$onFailure$1.apply(Future.scala:136)
      at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
      at org.spark_project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
      at scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:136)
      at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
      at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
      at scala.concurrent.Promise$class.tryFailure(Promise.scala:112)
      at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:153)
      at org.apache.spark.rpc.netty.NettyRpcEnv.org$apache$spark$rpc$netty$NettyRpcEnv$$onFailure$1(NettyRpcEnv.scala:205)
      at org.apache.spark.rpc.netty.NettyRpcEnv$$anon$1.run(NettyRpcEnv.scala:239)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:70)
      at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:174)
      at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:270)
      at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
  Caused by: org.apache.spark.rpc.RpcTimeoutException: Cannot receive any reply in 120 seconds. This timeout is controlled by spark.rpc.askTimeout
      at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48)
      at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63)
      at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
      at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
      at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:216)
      at scala.util.Try$.apply(Try.scala:192)
      at scala.util.Failure.recover(Try.scala:216)
      at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:326)
      at scala.concurrent.Future$$anonfun$recover$1.apply(Future.scala:326)
      at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
      at org.spark_project.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
      at scala.concurrent.impl.ExecutionContextImpl$$anon$1.execute(ExecutionContextImpl.scala:136)
      at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
      at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
      at scala.concurrent.Promise$class.complete(Promise.scala:55)
      at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
      at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237)
      at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237)
      at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
      at scala.concurrent.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:63)
      at scala.concurrent.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:78)
      at scala.concurrent.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:55)
      at scala.concurrent.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:55)
      at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
      at scala.concurrent.BatchingExecutor$Batch.run(BatchingExecutor.scala:54)
      at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:106)
      at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
      at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
      at scala.concurrent.Promise$class.tryFailure(Promise.scala:112)
      at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:153)
      at org.apache.spark.rpc.netty.NettyRpcEnv.org$apache$spark$rpc$netty$NettyRpcEnv$$onFailure$1(NettyRpcEnv.scala:205)
      at org.apache.spark.rpc.netty.NettyRpcEnv$$anon$1.run(NettyRpcEnv.scala:239)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
  Caused by: java.util.concurrent.TimeoutException: Cannot receive any reply in 120 seconds
      ... 8 more
  </code></pre>
  
  <p>I am using my vm's private IP for <code>SPARK_DRIVER_HOST, SPARK_LOCAL_IP, SPARK_MASTER_HOST</code> and public IP as <code>SPARK_PUBLIC_DNS</code> and to connect to the master. The master and workers are running on the same vm. And this exact setup is working on an ec2 instance. Any help would be appreciated.</p>
  
  <p><strong>UPDATE:</strong> I am able to run spark-shell normally from within the machine. The problem seems to be similar to <a href="https://stackoverflow.com/questions/28666183/submitting-jobs-to-spark-ec2-cluster-remotely">this</a> The executors can't interact with the driver, although I have the ports open on the vm. Is there a way to bind driver to the public IP of my instance/laptop?</p>