Is pickle not compatible with twisted?
Question: I have made 2 application: The client extract data from a sql server (10k
lines), and send every line pickled to a "collector" server via socket. The
server uses twisted (this is mandatory) and receive every line, unpikle it and
store the data in another sql server.

Everytime i start sending data from client to server, in the first 200 line
(everytime a different line) **the server** throws an exception: SOMETIMES it
something like:

    
    
    Traceback (most recent call last):
      File "collector2.py", line 81, in dataReceived
        self.count,account  = pickle.loads(data)
      File "/usr/lib/python2.6/pickle.py", line 1374, in loads
        return Unpickler(file).load()
      File "/usr/lib/python2.6/pickle.py", line 858, in load
        dispatch[key](self)
      File "/usr/lib/python2.6/pickle.py", line 1138, in load_pop
        del self.stack[-1]
    IndexError: list assignment index out of range
    

But it's NOT every time the same. Printing my exception i red: Exception: pop
from empty list Exception: list index out of range Exception: "'" Exception:
list assignment index out of range

Another strange errors is: File "/usr/lib/python2.6/pickle.py", line 1124, in
find_class **import**(module) exceptions.ImportError: No module named ond'

    
    
    for i in listaSAI:
        crm={}
        try:
            crm['uid']=i[0]
        except:
            crm['uid']=None
        try:
            crm['type_cond']=i[01]
        except:
            crm['type_cond']=None
        try:
            crm['population_id']=i[2]
        except:
            crm['population_id']=None
        try:
            crm['product_id']=i[3]
        except:
            crm['product_id']=None
        try:
            crm['a_id']=i[4]
        except:
            crm['a_id']=None
        try:
            crm['status']=i[5]
        except:
            crm['status']=None
        #time.sleep(0.001)
        serialized = pickle.dumps((count,crm))
        #print "sent num", count, crm
        s.sendall(serialized)
        count += 1
    

And my server:

    
    
    def dataReceived(self, data):
        try:
            self.count,account  = pickle.loads(data)        
        except Exception as e:
            print "Eccezione:", e
            print self.count+1
            print "DISAGIO", data
            print traceback.print_exc()
    

Printing the data in my client tells me that everything it's ok. _If i try to
slow down the process of sending using time.sleep(0.01) in my client,
EVERYTHING IS FINE, and no exception are raised._

What can i do to debug my code?

p.s. I suspect that exceptions.ImportError: No module named ond' refers to
"type_cond" key in crm.


Answer: Since you have no problem when adding a delay between writes, it is clear the
problem is not related to pickle, but to the transport of the data.

One option is that the client keeps writing data to the socket faster than the
server can process it, resulting in buffer overflow and packet loss.

Or, do you have more than one thread writing to the socket potentially at the
same time?

