title: Path to nested column not found in Spark DataFrame
body: <p>I'm having an issue and I'm not sure if Spark DataFrames are the problem or spark-xml which I'm using to parse an XML file into Spark. I would really appreciate any help.</p>
  
  <p>So, I have the following XML:</p>
  
  <pre><code>&lt;root&gt;
    &lt;path&gt;
      &lt;to&gt;
        &lt;atag&gt;
          &lt;atag_number&gt;1&lt;/atag_number&gt;
          &lt;more&gt;
            &lt;again&gt;
              &lt;text&gt;1111&lt;/text&gt;
            &lt;/again&gt;
          &lt;/more&gt;
          &lt;more&gt;
            &lt;again&gt;
              &lt;text&gt;2222&lt;/text&gt;
            &lt;/again&gt;
          &lt;/more&gt;
          &lt;more&gt;
            &lt;again&gt;
              &lt;text&gt;3333&lt;/text&gt;
            &lt;/again&gt;
          &lt;/more&gt;
        &lt;/atag&gt;
        &lt;atag&gt;
          &lt;atag_number&gt;2&lt;/atag_number&gt;
          &lt;more&gt;
            &lt;again&gt;
              &lt;text&gt;4444&lt;/text&gt;
            &lt;/again&gt;
          &lt;/more&gt;
          &lt;more&gt;
            &lt;again&gt;
              &lt;text&gt;5555&lt;/text&gt;
            &lt;/again&gt;
          &lt;/more&gt;
          &lt;more&gt;
            &lt;again&gt;
              &lt;text&gt;6666&lt;/text&gt;
            &lt;/again&gt;
          &lt;/more&gt;
        &lt;/atag&gt;
      &lt;/to&gt;
    &lt;/path&gt;
  &lt;/root&gt;
  </code></pre>
  
  <p>and I would like to get a table containing <code>path.to.atag.more.again.text</code>. I want them to be atomic, so it would need to be exploded to get a row for each <code>text</code> value.</p>
  
  <p>If I select e.g. <code>path.to.atag[0].more.again.text</code>, I get a list ['1111','2222','3333'].
  But if I want all a tags from the file, so if I select <code>path.to.atag.more.again.text</code>, I get an error, saying:</p>
  
  <pre><code>Traceback (most recent call last):
    File "...\spark-2.0.1-bin-hadoop2.7\python\pyspark\sql\utils.py", line 63, in deco
      return f(*a, **kw)
    File "...\spark-2.0.1-bin-hadoop2.7\python\lib\py4j-0.10.3-src.zip\py4j\protocol.py", line 319, in get_return_value
  py4j.protocol.Py4JJavaError: An error occurred while calling o34.selectExpr.
  : org.apache.spark.sql.AnalysisException: No such struct field text in again; line 1 pos 0
  
      at org.apache.spark.sql.catalyst.expressions.ExtractValue$.findField(complexTypeExtractors.scala:85)
  
      at org.apache.spark.sql.catalyst.expressions.ExtractValue$.apply(complexTypeExtractors.scala:58)
  
      at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$3.apply(LogicalPlan.scala:253)
  
      at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$3.apply(LogicalPlan.scala:252)
  
      at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
  
      at scala.collection.immutable.List.foldLeft(List.scala:84)
  
      at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolve(LogicalPlan.scala:252)
  
      at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveChildren(LogicalPlan.scala:148)
  
      at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9$$anonfun$applyOrElse$5$$anonfun$31.apply(Analyzer.scala:604)
  
      at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9$$anonfun$applyOrElse$5$$anonfun$31.apply(Analyzer.scala:604)
  
      at org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:48)
  
      at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9$$anonfun$applyOrElse$5.applyOrElse(Analyzer.scala:604)
  
      at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9$$anonfun$applyOrElse$5.applyOrElse(Analyzer.scala:600)
  
      at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:301)
  
      at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:301)
  
      at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
  
      at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:300)
  
      at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:298)
  
      at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:298)
  
      at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:321)
  
      at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:179)
  
      at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildren(TreeNode.scala:319)
  
      at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:298)
  
      at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionUp$1(QueryPlan.scala:191)
  
      at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$2(QueryPlan.scala:201)
  
      at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$2$1.apply(QueryPlan.scala:205)
  
      at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  
      at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  
      at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  
      at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
  
      at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
  
      at scala.collection.AbstractTraversable.map(Traversable.scala:104)
  
      at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$2(QueryPlan.scala:205)
  
      at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$5.apply(QueryPlan.scala:210)
  
      at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:179)
  
      at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsUp(QueryPlan.scala:210)
  
      at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9.applyOrElse(Analyzer.scala:600)
  
      at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$$anonfun$apply$9.applyOrElse(Analyzer.scala:542)
  
      at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:61)
  
      at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:61)
  
      at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
  
      at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:60)
  
      at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.apply(Analyzer.scala:542)
  
      at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.apply(Analyzer.scala:479)
  
      at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:85)
  
      at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:82)
  
      at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
  
      at scala.collection.immutable.List.foldLeft(List.scala:84)
  
      at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:82)
  
      at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:74)
  
      at scala.collection.immutable.List.foreach(List.scala:381)
  
      at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:74)
  
      at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:65)
  
      at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:63)
  
      at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:51)
  
      at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
  
      at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withPlan(Dataset.scala:2603)
  
      at org.apache.spark.sql.Dataset.select(Dataset.scala:969)
  
      at org.apache.spark.sql.Dataset.selectExpr(Dataset.scala:1004)
  
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  
      at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
  
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  
      at java.lang.reflect.Method.invoke(Unknown Source)
  
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
  
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
  
      at py4j.Gateway.invoke(Gateway.java:280)
  
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
  
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
  
      at py4j.GatewayConnection.run(GatewayConnection.java:214)
  
      at java.lang.Thread.run(Unknown Source)
  
  
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "...\MyModule.py", line 67, in &lt;module&gt;
      df_output = df.selectExpr('path.to.atag.more.again.text')
    File "...\spark-2.0.1-bin-hadoop2.7\python\pyspark\sql\dataframe.py", line 875, in selectExpr
      jdf = self._jdf.selectExpr(self._jseq(expr))
    File "...\spark-2.0.1-bin-hadoop2.7\python\lib\py4j-0.10.3-src.zip\py4j\java_gateway.py", line 1133, in __call__
    File "...\spark-2.0.1-bin-hadoop2.7\python\pyspark\sql\utils.py", line 69, in deco
      raise AnalysisException(s.split(': ', 1)[1], stackTrace)
  pyspark.sql.utils.AnalysisException: 'No such struct field text in again; line 1 pos 0'
  </code></pre>