title: Executing hdfs commands from inside scala script
body: <p>I'm trying to execute a <code>HDFS</code> specific command from inside the scala script being executed by Spark in cluster mode. Below the command:</p>
  
  <pre><code>val cmd = Seq("hdfs","dfs","-copyToLocal","/tmp/file.dat","/path/to/local")
  val result = cmd.!!
  </code></pre>
  
  <p>The job fails at this stage with the error as below:</p>
  
  <pre><code>java.io.FileNotFoundException: /var/run/cloudera-scm-agent/process/2087791-yarn-NODEMANAGER/log4j.properties (Permission denied)
          at java.io.FileInputStream.open0(Native Method)
          at java.io.FileInputStream.open(FileInputStream.java:195)
          at java.io.FileInputStream.&lt;init&gt;(FileInputStream.java:138)
          at java.io.FileInputStream.&lt;init&gt;(FileInputStream.java:93)
          at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
          at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
          at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
          at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
          at org.apache.log4j.LogManager.&lt;clinit&gt;(LogManager.java:127)
          at org.apache.log4j.Logger.getLogger(Logger.java:104)
          at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:262)
          at org.apache.commons.logging.impl.Log4JLogger.&lt;init&gt;(Log4JLogger.java:108)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  </code></pre>
  
  <p>However, when I run the same command separately in Spark shell, it executes just fine and the file is copied as well.</p>
  
  <pre><code>scala&gt; val cmd = Seq("hdfs","dfs","-copyToLocal","/tmp/file_landing_area/file.dat","/tmp/local_file_area")
  cmd: Seq[String] = List(hdfs, dfs, -copyToLocal, /tmp/file_landing_area/file.dat, /tmp/local_file_area)
  
  scala&gt; val result = cmd.!!
  result: String = ""
  </code></pre>
  
  <p>I don't understand the permission denied error. Although it displays as a <code>FileNotFoundException</code>. Totally confusing.</p>
  
  <p>Any ideas?</p>