title: Why do I sometimes get "item size exceeded" message when batch-writing into DynamoDB?
body: <p>I have a DynamoDB table whose items are long strings. These string are JSON messages converted into strings.</p>
  
  <p>Each JSON message has a unique id (<code>eId</code> below) associated with it that is used as the hash key.</p>
  
  <p>If a string is longer than certain max (<code>maxlen</code> below), I split it into pieces and create individual items for each piece and then I create a "parent" message that refers to those pieces. This way I am trying to avoid creating items bigger than the 64K limit.</p>
  
  <p>I collect items into batches of length 25 and then batch-write into the table.</p>
  
  <p>I am not using <code>maxlen = 30000</code>. Still from time to time, batch write throws the <code>Item size exceeds limit</code> exception. Here's the splitting code:</p>
  
  <pre><code>mStr = json.dumps(m,encoding='utf-8', ensure_ascii=False, sort_keys=True)
  lStr = len(mStr)
  if lStr &gt; maxlen:
      np = int(math.ceil(float(lStr)/maxlen))
      log.info("Splitting message with id=%s of len=%d into %d pieces of upto %d" % (eId, lStr, np, maxlen))
      start=0
      parentMessage = ""
      for ip in range(np):
         fin=(ip+1)*maxlen
         ss = mStr[start:fin]
         eId_ss = eId + ("-part%dof%d" % (ip+1, np))
         item_data = {"split" : "0", "message" : ss}
         item = createNewItem(table,eId_ss,item_data)
  
         csP = cPickle.dumps(item)
         szP = sys.getsizeof(csP)
         log.info("%s: [%d : %d] = %d chars, item size %d" % (eId_ss,start,fin,len(ss),szP))
  
         items.append(item)
         item_eIds.append(eId_ss)
  
         parentMessage += (eId_ss + ";")
         start=fin
  
         item_data = {"split" : "1", "message" : parentMessage}
         item = createNewItem(table,eId,item_data)
         items.append(item)
         item_eIds.append(eId)
  else:
  </code></pre>
  
  <p>What I noticed was that from time to time the line <code>sys.getsizeof(csP)</code> seems to produce values that are upto about 4 times larger than the length of the string piece <code>ss</code>. That does not happen often but I see it in the log. So I presume it happens from time to time that it generates an item that's bigger than 64K.</p>
  
  <p>I, of course, can keep lowering <code>maxlen</code>. But since I don't understand how large items get generated, I don't know what value to pick. Plus, lower values of <code>maxlen</code> mean a lot more messages get split which slows down my code.</p>
  
  <p>Questions:
  1. Why does it sometimes generate items that are much bigger than the strings they "wrap"?
  2. More importantly, is there a robust way of ensuring my items are not too big without having to lower <code>maxlen</code> to ridiculously low values?</p>
  
  <p>Thx </p>
  
  <p>aaaa</p>