

pysparkで下記のようなエラーが発生しています．

[code]

    stringJSONRDD = sc.parallelize((""" 
    { "id": "123",
        "name": "Katie",
        "age": 19,
        "eyeColor": "brown"
      }""",
       """{
        "id": "234",
        "name": "Michael",
        "age": 22,
        "eyeColor": "green"
      }""", 
      """{
        "id": "345",
        "name": "Simone",
        "age": 23,
        "eyeColor": "blue"
      }""")
    )
    
[/code]

を行い，RDDを定義した後

[code]

    swimmersJSON = spark.read.json(stringJSONRDD)
    
[/code]

でDataFrameに変換した後．

[code]

    spark.sql("select*from swimmersJSON").collect()
    
[/code]

を実行すると

[code]

    ---------------------------------------------------------------------------
    Py4JJavaError                             Traceback (most recent call last)
    ~/Spark/python/pyspark/sql/utils.py in deco(*a, **kw)
         62         try:
    ---> 63             return f(*a, **kw)
         64         except py4j.protocol.Py4JJavaError as e:
    
    ~/Spark/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
        318                     "An error occurred while calling {0}{1}{2}.\n".
    --> 319                     format(target_id, ".", name), value)
        320             else:
    
    Py4JJavaError: An error occurred while calling o23.sql.
    : org.apache.spark.sql.AnalysisException: Table or view not found: swimmersJSON; line 1 pos 12
        at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$lookupTableFromCatalog(Analyzer.scala:649)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.resolveRelation(Analyzer.scala:601)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:631)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:624)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:62)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:62)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:61)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$1.apply(LogicalPlan.scala:59)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$1.apply(LogicalPlan.scala:59)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:59)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:624)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:570)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:85)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:82)
        at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
        at scala.collection.immutable.List.foldLeft(List.scala:84)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:82)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:74)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:74)
        at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:69)
        at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:67)
        at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:50)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:67)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:632)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:280)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:214)
        at java.lang.Thread.run(Thread.java:748)
    
    
    During handling of the above exception, another exception occurred:
    
    AnalysisException                         Traceback (most recent call last)
    <ipython-input-8-6317c6e34aea> in <module>()
    ----> 1 spark.sql("select*from swimmersJSON").collect()
    
    ~/Spark/python/pyspark/sql/session.py in sql(self, sqlQuery)
        601         [Row(f1=1, f2=u'row1'), Row(f1=2, f2=u'row2'), Row(f1=3, f2=u'row3')]
        602         """
    --> 603         return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
        604 
        605     @since(2.0)
    
    ~/Spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py in __call__(self, *args)
       1131         answer = self.gateway_client.send_command(command)
       1132         return_value = get_return_value(
    -> 1133             answer, self.gateway_client, self.target_id, self.name)
       1134 
       1135         for temp_arg in temp_args:
    
    ~/Spark/python/pyspark/sql/utils.py in deco(*a, **kw)
         67                                              e.java_exception.getStackTrace()))
         68             if s.startswith('org.apache.spark.sql.AnalysisException: '):
    ---> 69                 raise AnalysisException(s.split(': ', 1)[1], stackTrace)
         70             if s.startswith('org.apache.spark.sql.catalyst.analysis'):
         71                 raise AnalysisException(s.split(': ', 1)[1], stackTrace)
    
    AnalysisException: 'Table or view not found: swimmersJSON; line 1 pos 12'
    
[/code]

といったエラーが出ます．どうすればよろしいでしょうか？  
なお

[code]

    spark.sql("show databases").show
    
[/code]

では

[code]

    <bound method DataFrame.show of DataFrame[databaseName: string]>
    
[/code]

と処理はされている様子です．

./build/mvn -DskipTests clean package  
のテストや  
./python/run-testは全部通っています．

Python 3.6.2 :: Anaconda, Inc.  
(3-5.0.0)  
java version "1.8.0_151"  
spark-2.2.1


CC BY-SA 3.0
2017-12-15T13:51:10.717
40311
2018-12-12T10:02:00.417
26670
question
python sql spark
pysparkでのspark.sqlに対して"Py4JJavaError: An error occurred while calling o23.sql."


データフレームを作っただけではテーブルにはなりません。次のようにしてテーブルを登録しましょう。

[code]

    swimmersJSON.createOrReplaceTempView("swimmersJSON")
    
[/code]

ところで次のように書かれていますが、これではshowは実行されていません。.showはメソッドだと言っているだけです。実行するには.show()としなければなりません。

[code]

    spark.sql("show databases").show
    <bound method DataFrame.show of DataFrame[databaseName: string]>
    
[/code]


CC BY-SA 3.0
2018-01-10T23:09:55.947
40872
2018-01-10T23:19:55.837
2018-01-10T23:19:55.837
7837
7837
40311
answer