Title: How to pass configuration file that hosted in HDFS to Spark Application?
Tags: apache-spark;hadoop;configuration;apache-spark-sql;spark-structured-streaming
Question: I'm working with Spark Structured Streaming. Also, I'm working with ```Scala```. I want to pass config file to my spark application. This configuration file hosted in ```HDFS```. For example; 

spark_job.conf (HOCON)

```spark {
  appName: "",
  master: "",
  shuffle.size: 4 
  etc..
}

kafkaSource {
  servers: "",
  topic: "",
  etc..
}

redisSink {
  host: "",
  port: 999,
  timeout: 2000,
  checkpointLocation: "hdfs location",
  etc..
}
```

How can I pass it to Spark Application? How can I read this file(```hosted HDFS```) in Spark?

Comment: I've seen this solution. But I'm looking other ways. Because this way isn't seen good to me. Also, I'm using Scala.

Comment: Because, this solution uses hadoop configuration files. My code have not "hadoop conf" files. Also, I want to pass any "spark conf" to application like appName etc. Thus I can't use spark.sparkContext.hadoopConfiguration. I don't know how can I do this.

Comment: I'm using typesafe api. I think you are right. The only way is seems this(read file from hdfs in the each worker node) for now.

Comment: Does https://stackoverflow.com/questions/17072543/reading-hdfs-and-local-files-in-java help?

Comment: There's also an equivalent Scala API. If for whatever reason you don't want to do it that way, you should explain why not.

Comment: If you don't want to read your data into a Hadoop Configuration object, https://github.com/lightbend/config is a popular generic HOCON configuration reader. You still need to read the file using the Scala HDFS API.

Comment: It is Serializable, so you can read the config once in the driver and have it passed to the task nodes.

Here is the accepted answer: You can read the HOCON config from HDFS in the following way:
```import com.typesafe.config.{Config, ConfigFactory}
import java.io.InputStreamReader
import java.net.URI
import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.hadoop.conf.Configuration

val hdfs: FileSystem = FileSystem.get(new URI(&quot;hdfs://&quot;), new Configuration())

val reader = new InputStreamReader(hdfs.open(new Path(&quot;/path/to/conf/on/hdfs&quot;)))

val conf: Config = ConfigFactory.parseReader(reader)
```
You can also pass the URI of your namenode to the ```FileSystem.get(new URI(&quot;your_uri_here&quot;))``` and the code will still read your configuration.

Comment for this answer: In the `new Path("/path/to/conf/on/hdfs")`,  should address be the `hdfs://path/to/conf` or just `path/to/conf` ? Also, thank you so much!
Comment for this answer: typo in import com.typesafe.config.{Cofig, ConfigFactory} .. should be "Config". This was too small of a change to edit in the post
Comment for this answer: only the path on hdfs without the `hdfs://` prefix. e.g /user/a-better-world/conf/spark_job.conf


{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 1, 'CommentCount': 7, 'ContentLicense': 'CC BY-SA 4.0', 'CreationDate': '2019:00:05/07/19', 'FavoriteCount': None, 'PostTypeId': 1, 'Tags': '<apache-spark><hadoop><configuration><apache-spark-sql><spark-structured-streaming>', 'Title': 'How to pass configuration file that hosted in HDFS to Spark Application?', 'ViewCount': 1990, 'id': 56021255, 'user_AboutMe': '', 'user_AccountId': 11408122.0, 'user_DisplayName': 'a-better-world', 'user_Reputation': 43.0, 'user_Views': 24.0, 'user_id': 8363034.0}}