title: Using spark to access HDFS failed
body: <p>I am using Cloudera 4.2.0 and Spark.</p>
  <p>I just want to try out some examples given by Spark.</p>
  <pre class="lang-scala prettyprint-override"><code>// HdfsTest.scala
  package spark.examples
  
  import spark._
  
  object HdfsTest {
    def main(args: Array[String]) {
      val sc = new SparkContext(args(0), &quot;HdfsTest&quot;,
        System.getenv(&quot;SPARK_HOME&quot;), Seq(System.getenv(&quot;SPARK_EXAMPLES_JAR&quot;)))
  
      val file = sc.textFile(&quot;hdfs://n1.example.com/user/cloudera/data/navi_test.csv&quot;)
      val mapped = file.map(s =&gt; s.length).cache()
      for (iter &lt;- 1 to 10) {
        val start = System.currentTimeMillis()
        for (x &lt;- mapped) { x + 2 }
        //  println(&quot;Processing: &quot; + x)
        val end = System.currentTimeMillis()
        println(&quot;Iteration &quot; + iter + &quot; took &quot; + (end-start) + &quot; ms&quot;)
      }
      System.exit(0)
    }
  }
  </code></pre>
  <p>It's ok for compiling, but there is always some runtime problems:</p>
  <pre><code>Exception in thread &quot;main&quot; java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.hdfs.HftpFileSystem could not be instantiated: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.fs.DelegationTokenRenewer.&lt;init&gt;(Ljava/lang/Class;)V from class org.apache.hadoop.hdfs.HftpFileSystem
      at java.util.ServiceLoader.fail(ServiceLoader.java:224)
      at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
      at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
      at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
      at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2229)
      at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2240)
      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2257)
      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:86)
      at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2296)
      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2278)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:316)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:162)
      at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:587)
      at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:315)
      at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:288)
      at spark.SparkContext.hadoopFile(SparkContext.scala:263)
      at spark.SparkContext.textFile(SparkContext.scala:235)
      at spark.examples.HdfsTest$.main(HdfsTest.scala:9)
      at spark.examples.HdfsTest.main(HdfsTest.scala)
  Caused by: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.fs.DelegationTokenRenewer.&lt;init&gt;(Ljava/lang/Class;)V from class org.apache.hadoop.hdfs.HftpFileSystem
      at org.apache.hadoop.hdfs.HftpFileSystem.&lt;clinit&gt;(HftpFileSystem.java:84)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
      at java.lang.Class.newInstance0(Class.java:374)
      at java.lang.Class.newInstance(Class.java:327)
      at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
  ... 16 more
  </code></pre>
  <p>I have searched on Google, no idea about this kind of exception for Spark and HDFS.</p>
  <p><code>val file = sc.textFile(&quot;hdfs://n1.example.com/user/cloudera/data/navi_test.csv&quot;)</code> is where the problem occurs.</p>
  <pre><code>13/04/04 12:20:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  </code></pre>
  <p>And I got this Warning. Maybe I should add some hadoop paths in CLASS_PATH.</p>
  <p>Feel free to give any clue. =)</p>
  <p>Thank you all.</p>
  <p>REN Hao</p>