title: How to return a Pandas.Series in a user defined function, aka udf?
body: <p>Before the question, I would like to point I don't have any knowledge on spark, I started messing with it last week in a project made by the research team. I have to do some adjust in their code and I'm have a hard time with <code>pandas_udf</code> functions. </p>
  
  <p>The research team code has a class with some methods and there is 5 <code>pandas_udf</code> methods outside of the class using global variables to manipulate data. The modification I'm doing is to take out those global variables and also get those 5 <code>pandas_udf</code> methods inside the class, and I got an error doing that.</p>
  
  <p>Here is the code:    </p>
  
  <pre><code>class Signal(object):
      # some methods, __init__, etc, etc
  
  
      def propagate(self, column1):
          signal_list = []
          # some code and return a list
          return signal_list
  
      @pandas_udf(ArrayType(StringType()),  PandasUDFType.SCALAR)
      def propagate_signal(self, signal_column):
          return signal_column.apply(
              self.propagate,
              column1=signal_column
          )
  
      def execute(self):
          # some code...
          dataframe = dataframe.withColumn('col_name', self.propagate_signal(dataframe['col_name']))
  </code></pre>
  
  <p>Here I was getting an error, I made some reaserach, compared both codes, executed them both at same time on debug mode and figured out that when the <code>pandas_udf</code> method was getting executed the self param was not being expected and was causing the error, while the original code the method doesn't have the self param and was being executed pretty well. Then I came with a idea, nested function.    </p>
  
  <pre><code>def propagate_signal(self, signal_column):
      @pandas_udf(ArrayType(StringType()), PandasUDFType.SCALAR)
      def propagate_signal_nested(signal_column):
          signal_column.apply(self.propagate, column1=signal_column)
  
      return = propagate_signal_nested(signal_column)
  </code></pre>
  
  <p>It worked (I think it worked because it passed from here, it was not passing from this line), but right after this line I can't convert this spark dataframe to a pandas dataframe (just to note, before this line I can call dataframe.toPandas() and it get executed and I can see the dataframe), when I call <code>dataframe.toPandas()</code> after that line I'm getting a error <code>TypeError: Return type of the user-defined function should be Pandas.Series, but is &lt;class 'NoneType'&gt;</code> and also getting the error <code>ERROR TaskSetManager: Task 0 in stage 5.0 failed 1 times; aborting job</code> which I tried to search and got some suggestions about <code>parallelize</code>, increase memory, but it didn't work.</p>
  
  <p>So an simple question here after this whole history, is this <code>pandas_udf</code> function correct, what I'm doing wrong with the return, how can I make it return a pandas.Series? If anyone could help, please. Thanks for your attention.</p>