title: Writing test for pyspark udf
body: <p>I am having some internal python dependency being executed inside a spark <code>pandas_udf</code> . To pass parameters we are wrapping this inside another function.</p>
  <p>Code looks like below</p>
  <pre><code>def wrapper_fn(df, parameters):
      @pandas_udf(schema,GROUPED_MAP)
      def run_pandas_code():
          &quot;&quot;&quot; Importing some python library and using it &quot;&quot;&quot;
          return pandas_df
      return df.groupby(&lt;key&gt;).apply(run_pandas_code)
  </code></pre>
  <p>I want to write a test that executes the function <code>wrapper_fn</code> . But when i write these test, i get pickle error . Can someone recommend a good way to test pyspark udf's .</p>