title: Unable to fix UnknownHostException while reading a csv file from a HDFS dir
body: <p>My spark program is running on a server: <code>serverA</code>. I am running the code from pyspark terminal. Using that program, I am trying to read a csv file from another cluster set up on another server -&gt; server: <code>serverB</code>, HDFS cluster: <code>clusterB</code> as below:</p>
  <pre><code> spark = SparkSession.builder.master('yarn').appName(&quot;Detector&quot;).config('spark.app.name','dummy_App').config('spark.executor.memory','2g').config('spark.executor.cores','2').config('spark.yarn.keytab','/home/testuser/testuser.keytab').config('spark.yarn.principal','krbtgt/HADOOP.NAME.COM@NAME.COM').config('spark.executor.instances','1').config('hadoop.security.authentication','kerberos').config('spark.yarn.access.hadoopFileSystems','hdfs://clusterB').config('spark.yarn.principal','testuser@NAME.COM').getOrCreate()
  </code></pre>
  <p>The file I am trying to read is on cluster: <code>clusterB</code> as below:</p>
  <pre><code>(base) testuser@hdptetl:[~] {46} $ hadoop fs -df -h
  Filesystem          Size     Used  Available  Use%
  hdfs://clusterB  787.3 T  554.5 T    230.7 T   70%
  </code></pre>
  <p>The keytab details (path of keytab, KDC REALM) I mentioned in the spark configuration is present on the server <code>serverB</code>
  When I try to load the file as:</p>
  <pre><code>csv_df = spark.read.format('csv').load('hdfs://botest01/test/mr/wc.txt')
  </code></pre>
  <p>The code results in <code>UnknownHostException</code> as below:</p>
  <pre><code>&gt;&gt;&gt; tdf = spark.read.format('csv').load('hdfs://clusterB/test/mr/wc.txt')
  20/07/15 15:40:36 WARN FileStreamSink: Error while looking for metadata directory.
  Traceback (most recent call last):
    File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
    File &quot;/usr/hdp/current/spark2-client/python/pyspark/sql/readwriter.py&quot;, line 166, in load
      return self._df(self._jreader.load(path))
    File &quot;/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py&quot;, line 1160, in __call__
    File &quot;/usr/hdp/current/spark2-client/python/pyspark/sql/utils.py&quot;, line 79, in deco
      raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
  pyspark.sql.utils.IllegalArgumentException: u'java.net.UnknownHostException: clusterB'
  </code></pre>
  <p>Could anyone let me know what is the mistake I did here and how can I fix it ?</p>