title: How to use pyspark to read a table with emr-dynamodb-connector
body: <p>I want to use pyspark with <a href="https://github.com/awslabs/emr-dynamodb-connector" rel="nofollow noreferrer">emr-dynamodb-connector</a> to read an entire dynamodb table into an rdd or, preferably, a dataframe. My code follows.</p>
  
  <p><code>dynamodb.py</code>:</p>
  
  <pre><code>from pyspark import SparkContext
  from pyspark.sql import SparkSession
  
  sc = SparkContext.getOrCreate()
  spark = SparkSession(sc)
  
  conf = {
      "dynamodb.servicename": "dynamodb",
      "dynamodb.input.tableName": "user_state_test",
      "dynamodb.output.tableName": "user_state_test",
      "dynamodb.endpoint": "https://dynamodb.us-west-2.amazonaws.com",
      "dynamodb.regionid": "us-west-2",
      "mapred.output.format.class": "org.apache.hadoop.dynamodb.write.DynamoDBOutputFormat",
      "mapred.input.format.class": "org.apache.hadoop.dynamodb.read.DynamoDBInputFormat"
  }
  
  dynamoRDD = sc.hadoopRDD(
      inputFormatClass="org.apache.hadoop.dynamodb.read.DynamoDBInputFormat",
      keyClass="org.apache.hadoop.io.Text",
      valueClass="org.apache.hadoop.dynamodb.DynamoDBItemWritable",
      conf=conf
  )
  
  count = dynamoRDD.count()
  print(count)
  </code></pre>
  
  <p>In order to supply the emr connector, I built it using the maven build tool in accordance with the <a href="https://github.com/awslabs/emr-dynamodb-connector#how-to-build" rel="nofollow noreferrer">awslabs instructions</a>: </p>
  
  <ol>
  <li>Clone repo</li>
  <li><code>mvn clean install</code>. </li>
  <li>The build results in a new jar in a target directory in the <code>emr-dynamodb-hadoop</code> dir of the repo, called <code>emr-dynamodb-hadoop-4.8.0-SNAPSHOT.jar</code>. I copied this jar into the repo my code is in and renamed it to <code>emr-dynamodb-hadoop.jar</code>.</li>
  </ol>
  
  <p>I submit my spark job with the following command:</p>
  
  <p><code>spark-submit --master "local[4]" --jars /Users/vaerk/dev/myproject/emr-dynamodb-hadoop.jar dynamodb.py</code></p>
  
  <p>This code results in the following exception: <code>java.lang.ClassNotFoundException: com.amazonaws.services.dynamodbv2.model.AttributeValue</code></p>
  
  <p>My questions:</p>
  
  <ul>
  <li>Is the <code>AttributeValue</code> class contained in the <code>emr-dynamodb-hadoop.jar</code>? The answers on <a href="https://stackoverflow.com/questions/46374369/local-spark-installation-not-working-as-expected">this question</a> indicate that is should be.</li>
  <li>Does my spark submit correctly load that jar onto the classpath?</li>
  <li>and of course, what do I need to do to get this to work?</li>
  </ul>