Title: Local or remote jar for Spark submit
Tags: java;apache-spark;hadoop;apache-kafka
Question: I have a spark-submit job running in yarn-client (driver on client side) that among other functions, reads and sends messages through Kafka.
On my cluster, only Kafka broker nodes have Kafka jars and I'd prefer not to made all my nodes Kafka clients. This means that my Spark job has to load (--jars) and copy Kafka jars from client server to all executors on the cluster.
My question is : what is the best practice in general ? Does copying those jars (~20MB) to each executor could have an impact on performance/latency ? or is it a classic way to work ?
Environment
Spark 1.6
Kafka 0.10.1
Network 10GB


Here is another answer: Best option, in general, is to make your code into a fat jar and just bundle all necessary dependencies as part of it.
If you find your submission process is too slow, feel free to copy the necessary Kafka Streaming JARs (including their dependencies) into the SPARK_HOME of each YARN NodeManager. I believe you can also reference an HDFS path to a JAR file, and if you increase the replication factor of that file, it'll amortize the network traffic
All your nodes need to be Kafka clients, regardless. You can't control where Spark starts an executor. Note: "installing Kafka" is not the same as making the Java client available on a server.
Comment for this answer: Thanks for your answer. As I use Ambari and because Kafka client is not managed by him, I would prefer not to deploy this client manually on all nodes (for maintenance purpose). I know that we can't control where Spark job will executed, that's why I'm asking myself if deploying Kafka jar from client side to executors on the cluster (at job execution, from Spark jetty server) is a good practice or not in my case.
Comment for this answer: Even if Kafka were managed by Ambari, the Kafka classpath contains no Spark libraries
Comment for this answer: Personally, I use Puppet or Ambari to maintain external systems behind Ambari. If you set that up, there's not that much to maintain

{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 0, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2018:00:04/10/18', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<java><apache-spark><hadoop><apache-kafka>', 'Title': 'Local or remote jar for Spark submit', 'ViewCount': 648, 'id': 49751878, 'user_AboutMe': None, 'user_AccountId': 4541003.0, 'user_DisplayName': 'tonio94', 'user_Reputation': 440.0, 'user_Views': 33.0, 'user_id': 3688882.0}}