title: How to test non-serializable code in Spark
body: <p>I have a spark code which sends requests to DynamoDB. The <a href="http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/dynamodbv2/AmazonDynamoDBClient.html" rel="nofollow noreferrer">AmazonDynamoDBClient</a>  which is used to make connection with database is not serializable. </p>
  
  <p>So I'm making instance of this class within <code>mapPartition</code> method in Scala like this -></p>
  
  <pre><code>recordsToWrite.mapPartitions { iter =&gt;
    var credentials = new BasicAWSCredentials(awsAccess, awsSecret)
    var client= new AmazonDynamoDBClient(credentials)
    var dynamoDB=new DynamoDB(client)
    var optTable=dynamoDB.getTable(tableName)
    iter.map { x =&gt; 
      //some code....
      optTable.updateItem(x)
    }
  }
  </code></pre>
  
  <p>The problem is I want to test this code with local spark (spark-testing-base) and dynamodb in unit test. </p>
  
  <p>I can not take <code>AmazonDynamoDBClient</code> out of <code>mapPartition</code> since its not serializable (Exception is thrown by spark)</p>