class_docstr: Spark related features. Usually, the features here are missing in pandas
  but Spark has it.
class_docstr_tok: ["Spark","related","features",".","Usually",",","the","features","here","are","missing","in","pandas","but","Spark","has","it","."]
docstr: Applies a function that takes and returns a Spark DataFrame. It allows natively
  apply a Spark function and column APIs with the Spark column internally used
  in Series or Index.
  
  .. note:: set `index_col` and keep the column named as so in the output Spark
      DataFrame to avoid using the default index to prevent performance penalty.
      If you omit `index_col`, it will use default index which is potentially
      expensive in general.
  
  .. note:: it will lose column labels. This is a synonym of
      ``func(psdf.to_spark(index_col)).pandas_api(index_col)``.
  
  Parameters
  ----------
  func : function
      Function to apply the function against the data by using Spark DataFrame.
  
  Returns
  -------
  DataFrame
  
  Raises
  ------
  ValueError : If the output from the function is not a Spark DataFrame.
  
  Examples
  --------
  >>> psdf = ps.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]}, columns=["a", "b"])
  >>> psdf
     a  b
  0  1  4
  1  2  5
  2  3  6
  
  >>> psdf.spark.apply(
  ...     lambda sdf: sdf.selectExpr("a + b as c", "index"), index_col="index")
  ... # doctest: +NORMALIZE_WHITESPACE
         c
  index
  0      5
  1      7
  2      9
  
  The case below ends up with using the default index, which should be avoided
  if possible.
  
  >>> psdf.spark.apply(lambda sdf: sdf.groupby("a").count().sort("a"))
     a  count
  0  1      1
  1  2      1
  2  3      1
docstr_tok: ["Applies","a","function","that","takes","and","returns","a","Spark","DataFrame",".","It","allows","natively","apply","a","Spark","function","and","column","APIs","with","the","Spark","column","internally","used","in","Series","or","Index",".","..","note",":",":","set","`","index_col","`","and","keep","the","column","named","as","so","in","the","output","Spark","DataFrame","to","avoid","using","the","default","index","to","prevent","performance","penalty",".","If","you","omit","`","index_col","`",",","it","will","use","default","index","which","is","potentially","expensive","in","general",".","..","note",":",":","it","will","lose","column","labels",".","This","is","a","synonym","of","``","func","(","psdf.to_spark","(","index_col",")",")",".pandas_api","(","index_col",")","``",".","Parameters","--","--","--","--","--","func",":","function","Function","to","apply","the","function","against","the","data","by","using","Spark","DataFrame",".","Returns","--","--","--","-","DataFrame","Raises","--","--","--","ValueError",":","If","the","output","from","the","function","is","not","a","Spark","DataFrame",".","Examples","--","--","--","--",">",">",">","psdf","=","ps.DataFrame","(","{","``","a","''",":","[","1",",","2",",","3","]",",","``","b","''",":","[","4",",","5",",","6","]","}",",","columns=","[","``","a","''",",","``","b","''","]",")",">",">",">","psdf","a","b","0","1","4","1","2","5","2","3","6",">",">",">","psdf.spark.apply","(","...","lambda","sdf",":","sdf.selectExpr","(","``","a","+","b","as","c","''",",","``","index","''",")",",","index_col=","''","index","''",")","...","#","doctest",":","+NORMALIZE_WHITESPACE","c","index","0","5","1","7","2","9","The","case","below","ends","up","with","using","the","default","index",",","which","should","be","avoided","if","possible",".",">",">",">","psdf.spark.apply","(","lambda","sdf",":","sdf.groupby","(","``","a","''",")",".count","(",")",".sort","(","``","a","''",")",")","a","count","0","1","1","1","2","1","2","3","1"]
code: def apply(
      self,
      func: Callable[[PySparkDataFrame], PySparkDataFrame],
      index_col: Optional[Union[str, List[str]]] = None,
  ) -> "ps.DataFrame":
      """
      Applies a function that takes and returns a Spark DataFrame. It allows natively
      apply a Spark function and column APIs with the Spark column internally used
      in Series or Index.
  
      .. note:: set `index_col` and keep the column named as so in the output Spark
          DataFrame to avoid using the default index to prevent performance penalty.
          If you omit `index_col`, it will use default index which is potentially
          expensive in general.
  
      .. note:: it will lose column labels. This is a synonym of
          ``func(psdf.to_spark(index_col)).pandas_api(index_col)``.
  
      Parameters
      ----------
      func : function
          Function to apply the function against the data by using Spark DataFrame.
  
      Returns
      -------
      DataFrame
  
      Raises
      ------
      ValueError : If the output from the function is not a Spark DataFrame.
  
      Examples
      --------
      >>> psdf = ps.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]}, columns=["a", "b"])
      >>> psdf
         a  b
      0  1  4
      1  2  5
      2  3  6
  
      >>> psdf.spark.apply(
      ...     lambda sdf: sdf.selectExpr("a + b as c", "index"), index_col="index")
      ... # doctest: +NORMALIZE_WHITESPACE
             c
      index
      0      5
      1      7
      2      9
  
      The case below ends up with using the default index, which should be avoided
      if possible.
  
      >>> psdf.spark.apply(lambda sdf: sdf.groupby("a").count().sort("a"))
         a  count
      0  1      1
      1  2      1
      2  3      1
      """
      output = func(self.frame(index_col))
      SparkDataFrame = get_dataframe_class()
      if not isinstance(output, SparkDataFrame):
          raise ValueError(
              "The output of the function [%s] should be of a "
              "pyspark.sql.DataFrame; however, got [%s]."
              % (func, type(output))
          )
      return output.pandas_api(index_col)
  
imports: from abc import ABCMeta, abstractmethod
  from typing import TYPE_CHECKING, Callable, Generic, List, Optional, Union
  from pyspark import StorageLevel
  from pyspark.sql import Column, DataFrame
  from pyspark.sql.types import DataType, StructType
  from pyspark.pandas._typing import IndexOpsLike
  from pyspark.pandas.internal import InternalField
  from pyspark.sql.utils import get_column_class, get_dataframe_class
label_desc: Use image node_id 11 for calling the SparkFrameMethods obj's underlying member method code with example usage: obj.apply(func, index_col) and returns: output