python SIP log file processing
Question: i have a sniff/log file for VoIP/SIP generated by python scapy in format time | src | srcport | dst | dstport | payload the sniff python script looks like this:
    
    
    ## Import Scapy module
    from scapy.all import *
    import sys
    
    sys.stdout = open('data.txt', 'w')
    
    pkts = sniff(filter="udp and port 5060 and not port 22", count=0,prn=lambda x:x.sprintf("%sent.time% | %IP.src% | %IP.sport% | %IP.dst% | %IP.dport% | Payload {Raw:%Raw.load%\n}"))
    

each packet in one line and each line can have different size depends on SIP
message type (Register, 200 OK, Invite, Notify and so on...)

What i would like to get from the file are fields `time, src, srcport, dst,
dstport` and from `Payload` type (just right after Payload) of `SIP message,
From, To, Call-iD, Contact` and the whole payload and then prepare these to
insert into MySQL database.

    
    
    1st msg:
    07:57:01.894990 | 192.168.1.10 | 5060 | 192.168.1.1 | 5060 | Payload 'INVITE sip:210@test-lab.org SIP/2.0\r\nVia: 
    SIP/2.0/UDP 192.168.1.10:5060;rport;branch=z9hG4bK-9cbb0ba8\r\nRoute: <sip:192.168.1.1:5060;lr>\r\nFrom: "test-311" <sip:311@test-lab.org>;tag=3d13bd6f\r\n
    To: <sip:210@test-lab.org>\r\nCall-ID: 21b0e2c755973976d6d06702ca33b32f@10.193.40.249\r\nCSeq: 1 INVITE\r\n
    Contact: "test-311" <sip:311@192.168.1.10:5060;transport=UDP>\r\nMax-Forwards: 70\r\n
    Supported: 100rel,replaces\r\nAllow: ACK, BYE, CANCEL, INFO, INVITE, OPTIONS, NOTIFY, PRACK, REFER, UPDATE, MESSAGE\r\nContent-Type: application/sdp\r\nContent-Length:   276\r\n\r\nv=0\r\no=- 3506863524 285638052 IN IP4 192.168.1.10\r\ns=-\r\nc=IN IP4 192.168.1.10\r\nt=0 0\r\nm=audio 8000 RTP/AVP 8 0 18 101\r\nc=IN IP4 192.168.1.10\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:18 G729/8000\r\na=rtpmap:101 telephone-event/8000\r\na=fmtp:101 0-15\r\na=ptime:20\r\n'
    
    2nd msg:
    07:57:01.902618 | 192.168.1.1 | 5060 | 192.168.1.10 | 5060 | Payload 'SIP/2.0 100 Trying\r\nVia: SIP/2.0/UDP 192.168.1.10:5060;received=192.168.1.10;branch=z9hG4bK-9cbb0ba8;rport=5060\r\nFrom: "test-311" <sip:+38551311@test-lab.org>;tag=3d13bd6f\r\nTo: <sip:210@test-lab.org>\r\nCall-ID: 21b0e2c755973976d6d06702ca33b32f@192.168.1.10\r\nCSeq: 1 INVITE\r\n\r\n'
    

I have tried to read line by line and split but I do not know how to split and
take data from payload part. Any help is more then welcome.


Answer: Well, you can enter the data into mysql straight from this program too; it
might very well be the easiest approach.

    
    
    from scapy.all import *
    import sys
    
    # connect to mysql 
    connection = ...
    
    def insert_into_mysql(packet):
        # now you can use packet.src, packet.sport, packet.dst, packet.dport, and
        # I believe packet['Raw'].load
        connection.execute(...)
    
        # to not print the packet
        return None
    
        # to print the packet
        return x.sprintf("%sent.time% | %IP.src% | %IP.sport% | %IP.dst% | %IP.dport% | Payload {Raw:%Raw.load%\n}"
    
    pkts = sniff(filter="udp and port 5060", count=0, store=0, prn=insert_into_mysql)
    

But if you need to use the existing log, I think you need to use:

    
    
    for line in open('log.txt'):
        sent_time, src, sport, dst, dport, payload = line.split(' | ', 6)
        payload = payload.replace('Payload ', '')
    
        # to get the unquoted payload, I'd guess (can't test SIP though)
        from ast import literal_eval
        payload = literal_eval(payload)
    

