class_docstr: Min-in for pandas map operations. Currently, only :class:`DataFrame`
  can use this class.
docstr: Maps an iterator of batches in the current :class:`DataFrame` using a Python native
  function that takes and outputs a pandas DataFrame, and returns the result as a
  :class:`DataFrame`.
  
  The function should take an iterator of `pandas.DataFrame`\s and return
  another iterator of `pandas.DataFrame`\s. All columns are passed
  together as an iterator of `pandas.DataFrame`\s to the function and the
  returned iterator of `pandas.DataFrame`\s are combined as a :class:`DataFrame`.
  Each `pandas.DataFrame` size can be controlled by
  `spark.sql.execution.arrow.maxRecordsPerBatch`. The size of the function's input and
  output can be different.
  
  .. versionadded:: 3.0.0
  
  .. versionchanged:: 3.4.0
      Supports Spark Connect.
  
  Parameters
  ----------
  func : function
      a Python native function that takes an iterator of `pandas.DataFrame`\s, and
      outputs an iterator of `pandas.DataFrame`\s.
  schema : :class:`pyspark.sql.types.DataType` or str
      the return type of the `func` in PySpark. The value can be either a
      :class:`pyspark.sql.types.DataType` object or a DDL-formatted type string.
  barrier : bool, optional, default False
      Use barrier mode execution.
  
      .. versionadded: 3.5.0
  
  Examples
  --------
  >>> df = spark.createDataFrame([(1, 21), (2, 30)], ("id", "age"))
  
  Filter rows with id equal to 1:
  
  >>> def filter_func(iterator):
  ...     for pdf in iterator:
  ...         yield pdf[pdf.id == 1]
  ...
  >>> df.mapInPandas(filter_func, df.schema).show()  # doctest: +SKIP
  +---+---+
  | id|age|
  +---+---+
  |  1| 21|
  +---+---+
  
  Compute the mean age for each id:
  
  >>> def mean_age(iterator):
  ...     for pdf in iterator:
  ...         yield pdf.groupby("id").mean().reset_index()
  ...
  >>> df.mapInPandas(mean_age, "id: bigint, age: double").show()  # doctest: +SKIP
  +---+----+
  | id| age|
  +---+----+
  |  1|21.0|
  |  2|30.0|
  +---+----+
  
  Add a new column with the double of the age:
  
  >>> def double_age(iterator):
  ...     for pdf in iterator:
  ...         pdf["double_age"] = pdf["age"] * 2
  ...         yield pdf
  ...
  >>> df.mapInPandas(
  ...     double_age, "id: bigint, age: bigint, double_age: bigint").show()  # doctest: +SKIP
  +---+---+----------+
  | id|age|double_age|
  +---+---+----------+
  |  1| 21|        42|
  |  2| 30|        60|
  +---+---+----------+
  
  Set ``barrier`` to ``True`` to force the ``mapInPandas`` stage running in the
  barrier mode, it ensures all Python workers in the stage will be
  launched concurrently.
  
  >>> df.mapInPandas(filter_func, df.schema, barrier=True).show()  # doctest: +SKIP
  +---+---+
  | id|age|
  +---+---+
  |  1| 21|
  +---+---+
  
  Notes
  -----
  This API is experimental
  
  See Also
  --------
  pyspark.sql.functions.pandas_udf
docstr_tok: ["Maps","an","iterator","of","batches","in","the","current",":","class",":","`","DataFrame","`","using","a","Python","native","function","that","takes","and","outputs","a","pandas","DataFrame",",","and","returns","the","result","as","a",":","class",":","`","DataFrame","`",".","The","function","should","take","an","iterator","of","`","pandas.DataFrame","`","\\s","and","return","another","iterator","of","`","pandas.DataFrame","`","\\s",".","All","columns","are","passed","together","as","an","iterator","of","`","pandas.DataFrame","`","\\s","to","the","function","and","the","returned","iterator","of","`","pandas.DataFrame","`","\\s","are","combined","as","a",":","class",":","`","DataFrame","`",".","Each","`","pandas.DataFrame","`","size","can","be","controlled","by","`","spark.sql.execution.arrow.maxRecordsPerBatch","`",".","The","size","of","the","function","'s","input","and","output","can","be","different",".","..","versionadded",":",":","3.0.0","..","versionchanged",":",":","3.4.0","Supports","Spark","Connect",".","Parameters","--","--","--","--","--","func",":","function","a","Python","native","function","that","takes","an","iterator","of","`","pandas.DataFrame","`","\\s",",","and","outputs","an","iterator","of","`","pandas.DataFrame","`","\\s",".","schema",":",":","class",":","`","pyspark.sql.types.DataType","`","or","str","the","return","type","of","the","`","func","`","in","PySpark",".","The","value","can","be","either","a",":","class",":","`","pyspark.sql.types.DataType","`","object","or","a","DDL-formatted","type","string",".","barrier",":","bool",",","optional",",","default","False","Use","barrier","mode","execution",".","..","versionadded",":","3.5.0","Examples","--","--","--","--",">",">",">","df","=","spark.createDataFrame","(","[","(","1",",","21",")",",","(","2",",","30",")","]",",","(","``","id","''",",","``","age","''",")",")","Filter","rows","with","id","equal","to","1",":",">",">",">","def","filter_func","(","iterator",")",":","...","for","pdf","in","iterator",":","...","yield","pdf","[","pdf.id","==","1","]","...",">",">",">","df.mapInPandas","(","filter_func",",","df.schema",")",".show","(",")","#","doctest",":","+SKIP","+","--","-+","--","-+","|","id|age|","+","--","-+","--","-+","|","1|","21|","+","--","-+","--","-+","Compute","the","mean","age","for","each","id",":",">",">",">","def","mean_age","(","iterator",")",":","...","for","pdf","in","iterator",":","...","yield","pdf.groupby","(","``","id","''",")",".mean","(",")",".reset_index","(",")","...",">",">",">","df.mapInPandas","(","mean_age",",","``","id",":","bigint",",","age",":","double","''",")",".show","(",")","#","doctest",":","+SKIP","+","--","-+","--","--","+","|","id|","age|","+","--","-+","--","--","+","|","1|21.0|","|","2|30.0|","+","--","-+","--","--","+","Add","a","new","column","with","the","double","of","the","age",":",">",">",">","def","double_age","(","iterator",")",":","...","for","pdf","in","iterator",":","...","pdf","[","``","double_age","''","]","=","pdf","[","``","age","''","]","*","2","...","yield","pdf","...",">",">",">","df.mapInPandas","(","...","double_age",",","``","id",":","bigint",",","age",":","bigint",",","double_age",":","bigint","''",")",".show","(",")","#","doctest",":","+SKIP","+","--","-+","--","-+","--","--","--","--","--","+","|","id|age|double_age|","+","--","-+","--","-+","--","--","--","--","--","+","|","1|","21|","42|","|","2|","30|","60|","+","--","-+","--","-+","--","--","--","--","--","+","Set","``","barrier","``","to","``","True","``","to","force","the","``","mapInPandas","``","stage","running","in","the","barrier","mode",",","it","ensures","all","Python","workers","in","the","stage","will","be","launched","concurrently",".",">",">",">","df.mapInPandas","(","filter_func",",","df.schema",",","barrier=True",")",".show","(",")","#","doctest",":","+SKIP","+","--","-+","--","-+","|","id|age|","+","--","-+","--","-+","|","1|","21|","+","--","-+","--","-+","Notes","--","--","-","This","API","is","experimental","See","Also","--","--","--","--","pyspark.sql.functions.pandas_udf"]
code: def mapInPandas(
      self,
      func: "PandasMapIterFunction",
      schema: Union[StructType, str],
      barrier: bool = False,
  ) -> "DataFrame":
      """
      Maps an iterator of batches in the current :class:`DataFrame` using a Python native
      function that takes and outputs a pandas DataFrame, and returns the result as a
      :class:`DataFrame`.
  
      The function should take an iterator of `pandas.DataFrame`\\s and return
      another iterator of `pandas.DataFrame`\\s. All columns are passed
      together as an iterator of `pandas.DataFrame`\\s to the function and the
      returned iterator of `pandas.DataFrame`\\s are combined as a :class:`DataFrame`.
      Each `pandas.DataFrame` size can be controlled by
      `spark.sql.execution.arrow.maxRecordsPerBatch`. The size of the function's input and
      output can be different.
  
      .. versionadded:: 3.0.0
  
      .. versionchanged:: 3.4.0
          Supports Spark Connect.
  
      Parameters
      ----------
      func : function
          a Python native function that takes an iterator of `pandas.DataFrame`\\s, and
          outputs an iterator of `pandas.DataFrame`\\s.
      schema : :class:`pyspark.sql.types.DataType` or str
          the return type of the `func` in PySpark. The value can be either a
          :class:`pyspark.sql.types.DataType` object or a DDL-formatted type string.
      barrier : bool, optional, default False
          Use barrier mode execution.
  
          .. versionadded: 3.5.0
  
      Examples
      --------
      >>> df = spark.createDataFrame([(1, 21), (2, 30)], ("id", "age"))
  
      Filter rows with id equal to 1:
  
      >>> def filter_func(iterator):
      ...     for pdf in iterator:
      ...         yield pdf[pdf.id == 1]
      ...
      >>> df.mapInPandas(filter_func, df.schema).show()  # doctest: +SKIP
      +---+---+
      | id|age|
      +---+---+
      |  1| 21|
      +---+---+
  
      Compute the mean age for each id:
  
      >>> def mean_age(iterator):
      ...     for pdf in iterator:
      ...         yield pdf.groupby("id").mean().reset_index()
      ...
      >>> df.mapInPandas(mean_age, "id: bigint, age: double").show()  # doctest: +SKIP
      +---+----+
      | id| age|
      +---+----+
      |  1|21.0|
      |  2|30.0|
      +---+----+
  
      Add a new column with the double of the age:
  
      >>> def double_age(iterator):
      ...     for pdf in iterator:
      ...         pdf["double_age"] = pdf["age"] * 2
      ...         yield pdf
      ...
      >>> df.mapInPandas(
      ...     double_age, "id: bigint, age: bigint, double_age: bigint").show()  # doctest: +SKIP
      +---+---+----------+
      | id|age|double_age|
      +---+---+----------+
      |  1| 21|        42|
      |  2| 30|        60|
      +---+---+----------+
  
      Set ``barrier`` to ``True`` to force the ``mapInPandas`` stage running in the
      barrier mode, it ensures all Python workers in the stage will be
      launched concurrently.
  
      >>> df.mapInPandas(filter_func, df.schema, barrier=True).show()  # doctest: +SKIP
      +---+---+
      | id|age|
      +---+---+
      |  1| 21|
      +---+---+
  
      Notes
      -----
      This API is experimental
  
      See Also
      --------
      pyspark.sql.functions.pandas_udf
      """
      from pyspark.sql import DataFrame
      from pyspark.sql.pandas.functions import pandas_udf
  
      assert isinstance(self, DataFrame)
  
      # The usage of the pandas_udf is internal so type checking is disabled.
      udf = pandas_udf(
          func,
          returnType=schema,
          functionType=PythonEvalType.SQL_MAP_PANDAS_ITER_UDF,
      )  # type: ignore[call-overload]
      udf_column = udf(*[self[col] for col in self.columns])
      jdf = self._jdf.mapInPandas(udf_column._jc.expr(), barrier)
      return DataFrame(jdf, self.sparkSession)
  
file_path: spark/python/pyspark/sql/pandas/map_ops.py
imports: import sys
  from typing import Union, TYPE_CHECKING
  from pyspark.rdd import PythonEvalType
  from pyspark.sql.types import StructType
label_desc: Use image node_id 1 for calling the PandasMapOpsMixin obj's underlying member method code with example usage: obj.mapInPandas(func, schema, barrier) and returns: DataFrame