title: Run HiveFromSpark example with MASTER=yarn-cluster
body: <p>I'm trying to run <a href="https://github.com/apache/spark/blob/v1.3.0/examples/src/main/scala/org/apache/spark/examples/sql/hive/HiveFromSpark.scala" rel="nofollow">HiveFromSpark</a> example on my EMR Spark/Hive cluster.</p>
  
  <h2>The Problem</h2>
  
  <p>Using <a href="http://spark.apache.org/docs/latest/running-on-yarn.html" rel="nofollow">yarn-client</a>:</p>
  
  <pre><code>~/spark/bin/spark-submit --master yarn-client --num-executors=19 --class org.apache.spark.examples.sql.hive.HiveFromSpark  ~/spark/lib/spark-examples-1.3.0-hadoop2.4.0.jar
  </code></pre>
  
  <p>works like a charm. But, using <a href="http://spark.apache.org/docs/latest/running-on-yarn.html" rel="nofollow">yarn-cluster</a>:</p>
  
  <pre><code>~/spark/bin/spark-submit --master yarn-cluster --num-executors=19 --class org.apache.spark.examples.sql.hive.HiveFromSpark ~/spark/lib/spark-examples-1.3.0-hadoop2.4.0.jar
  </code></pre>
  
  <p>fails with:</p>
  
  <pre><code>Exception in thread "main" org.apache.spark.SparkException: Application finished with failed status
      at org.apache.spark.deploy.yarn.Client.run(Client.scala:622)
      at org.apache.spark.deploy.yarn.Client$.main(Client.scala:647)
      at org.apache.spark.deploy.yarn.Client.main(Client.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  </code></pre>
  
  <p>and the error message in Resource Manager says:</p>
  
  <pre><code>User class threw exception: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
  </code></pre>
  
  <p>Full log:</p>
  
  <pre><code>15/04/15 12:02:14 WARN DataNucleus.Persistence: Error creating validator of type org.datanucleus.properties.CorePropertyValidator
  ClassLoaderResolver for class "" gave error on creation : {1}
  org.datanucleus.exceptions.NucleusUserException: ClassLoaderResolver for class "" gave error on creation : {1}
      at org.datanucleus.NucleusContext.getClassLoaderResolver(NucleusContext.java:1087)
      at org.datanucleus.PersistenceConfiguration.validatePropertyValue(PersistenceConfiguration.java:797)
      at org.datanucleus.PersistenceConfiguration.setProperty(PersistenceConfiguration.java:714)
      at org.datanucleus.PersistenceConfiguration.setPersistenceProperties(PersistenceConfiguration.java:693)
      at org.datanucleus.NucleusContext.&lt;init&gt;(NucleusContext.java:273)
      at org.datanucleus.NucleusContext.&lt;init&gt;(NucleusContext.java:247)
      at org.datanucleus.NucleusContext.&lt;init&gt;(NucleusContext.java:225)
      at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.&lt;init&gt;(JDOPersistenceManagerFactory.java:416)
      at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:301)
      at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
      at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
      at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
      at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
      at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:310)
      at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:339)
      at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:248)
      at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:223)
      at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
      at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
      at org.apache.hadoop.hive.metastore.RawStoreProxy.&lt;init&gt;(RawStoreProxy.java:58)
      at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:497)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:475)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.&lt;init&gt;(HiveMetaStore.java:356)
      at org.apache.hadoop.hive.metastore.RetryingHMSHandler.&lt;init&gt;(RetryingHMSHandler.java:54)
      at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
      at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
      at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.&lt;init&gt;(HiveMetaStoreClient.java:171)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
      at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.&lt;init&gt;(RetryingMetaStoreClient.java:62)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
      at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
      at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
      at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
      at org.apache.spark.sql.hive.HiveContext.sessionState$lzycompute(HiveContext.scala:229)
      at org.apache.spark.sql.hive.HiveContext.sessionState(HiveContext.scala:225)
      at org.apache.spark.sql.hive.HiveContext.hiveconf$lzycompute(HiveContext.scala:241)
      at org.apache.spark.sql.hive.HiveContext.hiveconf(HiveContext.scala:240)
      at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:86)
      at org.apache.spark.examples.sql.hive.HiveFromSpark$.main(HiveFromSpark.scala:49)
      at org.apache.spark.examples.sql.hive.HiveFromSpark.main(HiveFromSpark.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480)
  Caused by: java.lang.NullPointerException
      at org.datanucleus.NucleusContext.getClassLoaderResolver(NucleusContext.java:1079)
      ... 58 more
  Nested Throwables StackTrace:
  java.lang.NullPointerException
      at org.datanucleus.NucleusContext.getClassLoaderResolver(NucleusContext.java:1079)
      at org.datanucleus.PersistenceConfiguration.validatePropertyValue(PersistenceConfiguration.java:797)
      at org.datanucleus.PersistenceConfiguration.setProperty(PersistenceConfiguration.java:714)
      at org.datanucleus.PersistenceConfiguration.setPersistenceProperties(PersistenceConfiguration.java:693)
      at org.datanucleus.NucleusContext.&lt;init&gt;(NucleusContext.java:273)
      at org.datanucleus.NucleusContext.&lt;init&gt;(NucleusContext.java:247)
      at org.datanucleus.NucleusContext.&lt;init&gt;(NucleusContext.java:225)
      at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.&lt;init&gt;(JDOPersistenceManagerFactory.java:416)
      at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:301)
      at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
      at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
      at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
      at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
      at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:310)
      at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:339)
      at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:248)
      at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:223)
      at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
      at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
      at org.apache.hadoop.hive.metastore.RawStoreProxy.&lt;init&gt;(RawStoreProxy.java:58)
      at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:497)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:475)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.&lt;init&gt;(HiveMetaStore.java:356)
      at org.apache.hadoop.hive.metastore.RetryingHMSHandler.&lt;init&gt;(RetryingHMSHandler.java:54)
      at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
      at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
      at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.&lt;init&gt;(HiveMetaStoreClient.java:171)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
      at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.&lt;init&gt;(RetryingMetaStoreClient.java:62)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
      at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
      at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
      at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
      at org.apache.spark.sql.hive.HiveContext.sessionState$lzycompute(HiveContext.scala:229)
      at org.apache.spark.sql.hive.HiveContext.sessionState(HiveContext.scala:225)
      at org.apache.spark.sql.hive.HiveContext.hiveconf$lzycompute(HiveContext.scala:241)
      at org.apache.spark.sql.hive.HiveContext.hiveconf(HiveContext.scala:240)
      at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:86)
      at org.apache.spark.examples.sql.hive.HiveFromSpark$.main(HiveFromSpark.scala:49)
      at org.apache.spark.examples.sql.hive.HiveFromSpark.main(HiveFromSpark.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480)
  15/04/15 12:02:14 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.    HiveMetaStoreClient
  java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
      at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
      at org.apache.spark.sql.hive.HiveContext.sessionState$lzycompute(HiveContext.scala:229)
      at org.apache.spark.sql.hive.HiveContext.sessionState(HiveContext.scala:225)
      at org.apache.spark.sql.hive.HiveContext.hiveconf$lzycompute(HiveContext.scala:241)
      at org.apache.spark.sql.hive.HiveContext.hiveconf(HiveContext.scala:240)
      at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:86)
      at org.apache.spark.examples.sql.hive.HiveFromSpark$.main(HiveFromSpark.scala:49)
      at org.apache.spark.examples.sql.hive.HiveFromSpark.main(HiveFromSpark.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480)
  Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
      at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.&lt;init&gt;(RetryingMetaStoreClient.java:62)
      at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
      at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
      at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
      at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
      ... 12 more
  Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
      at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
      ... 17 more
  Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception caught.
  NestedThrowables:
  java.lang.reflect.InvocationTargetException
      at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1193)
      at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
      at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
      at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:310)
      at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:339)
      at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:248)
      at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:223)
      at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
      at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
      at org.apache.hadoop.hive.metastore.RawStoreProxy.&lt;init&gt;(RawStoreProxy.java:58)
      at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:497)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:475)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
      at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.&lt;init&gt;(HiveMetaStore.java:356)
      at org.apache.hadoop.hive.metastore.RetryingHMSHandler.&lt;init&gt;(RetryingHMSHandler.java:54)
      at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
      at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
      at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.&lt;init&gt;(HiveMetaStoreClient.java:171)
      ... 22 more
  Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
      at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
      ... 41 more
  Caused by: org.datanucleus.exceptions.NucleusUserException: Persistence process has been specified to use a ClassLoaderResolver of name "datanucleus" yet this has   not   been found by the DataNucleus plugin mechanism. Please check your CLASSPATH and plugin specification.
      at org.datanucleus.NucleusContext.&lt;init&gt;(NucleusContext.java:283)
      at org.datanucleus.NucleusContext.&lt;init&gt;(NucleusContext.java:247)
      at org.datanucleus.NucleusContext.&lt;init&gt;(NucleusContext.java:225)
      at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.&lt;init&gt;(JDOPersistenceManagerFactory.java:416)
      at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:301)
      at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
      ... 49 more
  15/04/15 12:02:14 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.RuntimeException: Unable to     instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient)
  </code></pre>
  
  <h2>Cluster Setup</h2>
  
  <h3>EMR config</h3>
  
  <p>I setup the cluster with <code>Install Spark</code> (s3://support.elasticmapreduce/spark/install-spark) bootstrap action and <code>Setup Hive</code> step.</p>
  
  <pre><code>JAR location: s3://elasticmapreduce/libs/script-runner/script-runner.jar
  Main class: None
  Arguments: s3://elasticmapreduce/libs/hive/hive-script --base-path s3://elasticmapreduce/libs/hive/ --install-hive --hive-versions 0.13.1
  </code></pre>
  
  <h3>Software versions</h3>
  
  <p><code>hadoop version</code></p>
  
  <pre><code>Hadoop 2.4.0-amzn-3
  Subversion http://aws.amazon.com/elasticmapreduce/ -r d99c64d21bf338dcdd524bd08d7c393b6d1ac1d8
  Compiled by Elastic MapReduce on 2015-02-11T20:22Z
  Compiled with protoc 2.5.0
  From source with checksum 6c725ed23b3ecb95921fe461587fccf
  This command was run using /home/hadoop/.versions/2.4.0-amzn-3/share/hadoop/common/hadoop-common-2.4.0-amzn-3.jar
  </code></pre>
  
  <p>Spark version</p>
  
  <pre><code>Spark 1.3.0 built for Hadoop 2.4.0
  Build flags: -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests -Pkinesis-asl -Pspark-ganglia-lgpl -Phadoop-provided -Phive -Phive-thriftserver
  </code></pre>