title: aws emr pyspark stuck on collect call
body: <p>I am trying to learn to setup pyspark on aws emr. However, the sample job I am running is stuck in collect api call. I am using emr version 5-30.1.
  I dont see any relevant logs associated with this. May be, I am not looking at the right place.</p>
  <p>Following is the spark-submit command</p>
  <pre><code>spark-submit --deploy-mode cluster --master yarn --conf spark.yarn.maxAppAttempts=1 --conf spark.driver.memory=4g --conf spark.dynamicAllocation.enabled=true --conf spark.driver.cores=2 --conf spark.logConf=true --py-files s3://cdk_assets/assets/cdk.out/asset.5a391.zip s3://cdk_assets/assets/cdk.out/asset.3ab5bae.py'
  </code></pre>
  <p>This is the sample script I am running</p>
  <pre><code>from pyspark.sql import SparkSession
  
  def run():
      print(&quot;Sample started&quot;)
      spark_session = SparkSession.builder \
          .appName(&quot;Spark_Sample&quot;) \
          .master(&quot;yarn&quot;) \
          .getOrCreate()
      temp_c = [10, 3, -5, 25, 1, 9, 29, -10, 5]
      rdd_temp_c = spark_session.sparkContext.parallelize(temp_c)
      rdd_temp_k = rdd_temp_c.map(lambda x: x + 273.15).collect()
      print(rdd_temp_k)
  
  if __name__ == &quot;__main__&quot;:
      run()
  </code></pre>
  <p>I could see the following exceptions but I am not sure how they are related.
  First one is under the node//provision-node/apps-phase//install.stderr</p>
  <pre><code>2020-09-28 15:30:37,496 ERROR main: Encountered a problem while provisioning
  java.net.ConnectException: Connection refused (Connection refused)
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
      at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
      at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
      at java.net.Socket.connect(Socket.java:607)
      at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
      at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
      at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
      at sun.net.www.http.HttpClient.&lt;init&gt;(HttpClient.java:242)
      at sun.net.www.http.HttpClient.New(HttpClient.java:339)
      at sun.net.www.http.HttpClient.New(HttpClient.java:357)
      at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
      at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
      at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
      at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
      at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:93)
      at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972)
      at com.amazonaws.emr.node.provisioner.http.JsonHttpClient.doRequest(JsonHttpClient.java:49)
      at com.amazonaws.emr.node.provisioner.platform.EmrPlatformClient.getConfiguration(EmrPlatformClient.java:51)
      at com.amazonaws.emr.node.provisioner.platform.EmrPlatformClient.getConfiguration(EmrPlatformClient.java:43)
      at com.amazonaws.emr.node.provisioner.bigtop.config.PlatformContextProvider.provide(PlatformContextProvider.java:39)
      at com.amazonaws.emr.node.provisioner.workflow.NodeProvisionerWorkflow.work(NodeProvisionerWorkflow.java:97)
      at com.amazonaws.emr.node.provisioner.Program.main(Program.java:30)
  </code></pre>
  <p>Second is from yarn node-manager logs</p>
  <pre><code>2020-09-28 15:34:37,249 INFO org.apache.hadoop.ipc.Client (main): Retrying connect to server: ip-10-0-217-37.us-west-2.compute.internal/10.0.217.37:8025. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  2020-09-28 15:34:38,250 INFO org.apache.hadoop.ipc.Client (main): Retrying connect to server: ip-10-0-217-37.us-west-2.compute.internal/10.0.217.37:8025. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  2020-09-28 15:34:39,251 INFO org.apache.hadoop.ipc.Client (main): Retrying connect to server: ip-10-0-217-37.us-west-2.compute.internal/10.0.217.37:8025. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
  2020-09-28 15:34:39,252 WARN org.apache.hadoop.ipc.Client (main): Failed to connect to server: ip-10-0-217-37.us-west-2.compute.internal/10.0.217.37:8025: retries get failed due to exceeded maximum allowed retries number: 10
  java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:685)
      at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:788)
      at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:410)
      at org.apache.hadoop.ipc.Client.getConnection(Client.java:1550)
      at org.apache.hadoop.ipc.Client.call(Client.java:1381)
      at org.apache.hadoop.ipc.Client.call(Client.java:1345)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
      at com.sun.proxy.$Proxy79.registerNodeManager(Unknown Source)
      at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:73)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
      at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
      at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
      at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
      at com.sun.proxy.$Proxy80.registerNodeManager(Unknown Source)
      at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:344)
      at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:232)
      at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
      at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
      at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:369)
      at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
      at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:637)
      at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:684)
  2020-09-28 15:35:09,840 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager (main): Rolling master-key for container-tokens, got key with id 1723617652
  2020-09-28 15:35:09,845 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM (main): Rolling master-key for container-tokens, got key with id 1930837132
  2020-09-28 15:35:09,845 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl (main): NodeLabels sent from NM while registration were rejected by RM. And with message Node Labels {CORE} reported from NM with ID ip-10-0-192-171.us-west-2.compute.internal:8041 was rejected from RM with exception message as : Not all labels being replaced contained by known label collections, please check, new labels=[CORE]
  2020-09-28 15:35:09,845 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl (main): Registered with ResourceManager as ip-10-0-192-171.us-west-2.compute.internal:8041 with total resource of &lt;memory:6144, vCores:4&gt;
  2020-09-28 15:35:09,846 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl (main): Notifying ContainerManager to unblock new container-requests
  2020-09-28 15:35:09,938 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl (Node Status Updater): NM node labels {&lt;CORE:exclusivity=true&gt;} were not accepted by RM and message from RM : Node Labels {CORE} reported from NM with ID ip-10-0-192-171.us-west-2.compute.internal:8041 was rejected from RM with exception message as : Not all labels being replaced contained by known label collections, please check, new labels=[CORE]
  </code></pre>
  <p>Can somebody guide me with potential pointers on how to debug this issue further.
  Thanks in advance.</p>