Build/Parse an XML document from a socketstream in Python
Question: I have an issue parsing a continuous stream of (multiple) xml documents sent
by a third party over a socket. A sample of the xml stream sent over the
socket is:

    
    
    <?xml version="1.0"?><event><user id="1098"/><viewpage>109958</viewpage></event>
    <?xml version="1.0"?><event><user id="1482"/><actions><edit>102865</edit><commit>1592356</commit></actions></event>
    etc.
    

Here's the code I'm using:

    
    
    import socket
    import xml.etree.cElementTree as etree
    from StringIO import StringIO
    
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    host = "IP.IP.IP.IP"
    port = "8080"
    addr = (host,port)
    s.connect(addr)
    
    def iparse(packet):
        for _, element in etree.iterparse(packet):
            print ("%s, %s" %(element.tag, element.text))
            element.clear()
        #if complete <event> node received, publish node
    
    data = "<feeds>"
    while 1:
        chunk = s.recv(1024)
        #replace the xml doc declarations as comments
        data += (chunk.replace("<?","<!--")).replace("?>","-->")
        iparse(StringIO(data))
    

Things work just fine...however, the `for` loop in `iparse` iterates through
the entire doc each time. Is it possible for `iparse` to build and iterate
through one well-formed tag node (event) as it appears over the stream
instead? Note that there is no way in which I can set the chunk size to read a
well-formed packet. I could use a buffer, then build the packet and only send
to `iparse` once the packet is well-formed, but that would likely introduce
unwanted latency? Is there a better way to handle this?

**EDIT:**

Each event is distinct but contains arbitrary nodes under the root `<event>`.
`iparse` is expected to publish the latest event to an arbitrary number of
subscribers within a real-time analytics graphing system.


Answer: You could have a look at the [feed parsing](http://lxml.de/parsing.html#the-
feed-parser-interface) in `lxml.etree`. However, you're still going to run
into problems as your document is continually growing.

Are the XML blobs separated by new line characters? If so I suggest that you
buffer until you hit a new line and then send each line to an xml parser. Á la
[Twisted's
LineReceiver](http://twistedmatrix.com/trac/browser/tags/releases/twisted-11.0.0/twisted/protocols/basic.py#L509).

Actually, if it was me, I'd probably write this application in
[Twisted](http://twistedmatrix.com/trac/). Gluing together network services is
a common use case for it for me.

