Optimize Python Script to parse xml
Question: I'm parsing the US Patent XML files (downloaded from [Google patent
dumps](https://www.google.com/googlebooks/uspto-patents-redbook.html)) using
Python and Beautifulsoup; parsed data is exported to MYSQL database.

Each year's data contains close to 200-300K patents - which means parsing
200-300K xml files.

The server on which I'm running the python script is pretty powerful - 16
cores, 160 gigs of RAM, etc. but still it is taking close to 3 days to parse
one year's worth of data. [![enter image description
here](http://i.stack.imgur.com/Rve8M.png)](http://i.stack.imgur.com/Rve8M.png)
[![enter image description
here](http://i.stack.imgur.com/dEmQB.png)](http://i.stack.imgur.com/dEmQB.png)

I've been learning and using python since 2 years - so I can get stuff done
but do not know how to get it done in the most efficient manner. I'm reading
on it.

How can I optimize the below script to make it efficient?

Any guidance would be greatly appreciated.

Below is the code:

    
    
    from bs4 import BeautifulSoup
    import pandas as pd
    from pandas.core.frame import DataFrame
    import MySQLdb as db
    import os
    
    cnxn = db.connect('xx.xx.xx.xx','xxxxx','xxxxx','xxxx',charset='utf8',use_unicode=True)
    
    def separated_xml(infile):
        file = open(infile, "r")
        buffer = [file.readline()]
        for line in file:
            if line.startswith("<?xml "):
                yield "".join(buffer)
                buffer = []
            buffer.append(line)
        yield "".join(buffer)
        file.close()
    
    def get_data(soup):
        df = pd.DataFrame(columns = ['doc_id','patcit_num','patcit_document_id_country', 'patcit_document_id_doc_number','patcit_document_id_kind','patcit_document_id_name','patcit_document_id_date','category'])
        if soup.findAll('us-citation'):
            cit = soup.findAll('us-citation')
        else:
            cit = soup.findAll('citation')
        doc_id = soup.findAll('publication-reference')[0].find('doc-number').text
        for x in cit:
            try:
                patcit_num = x.find('patcit')['num']
            except:
                patcit_num = None
            try:
                patcit_document_id_country = x.find('country').text
            except:
                patcit_document_id_country = None   
            try:     
                patcit_document_id_doc_number = x.find('doc-number').text
            except: 
                patcit_document_id_doc_number = None
            try:
                patcit_document_id_kind = x.find('kind').text
            except:
                patcit_document_id_kind = None
            try:
                patcit_document_id_name = x.find('name').text
            except:
                patcit_document_id_name = None
            try: 
                patcit_document_id_date = x.find('date').text
            except:
                patcit_document_id_date = None
            try:
                category = x.find('category').text
            except:
                category = None
            print doc_id
            val = {'doc_id':doc_id,'patcit_num':patcit_num, 'patcit_document_id_country':patcit_document_id_country,'patcit_document_id_doc_number':patcit_document_id_doc_number, 'patcit_document_id_kind':patcit_document_id_kind,'patcit_document_id_name':patcit_document_id_name,'patcit_document_id_date':patcit_document_id_date,'category':category}    
            df = df.append(val, ignore_index=True)
        df.to_sql(name = 'table_name', con = cnxn, flavor='mysql', if_exists='append')
        print '1 doc exported'
    
    i=0
    
    l = os.listdir('/path/')
    for item in l:
        f = '/path/'+item
        print 'Currently parsing - ',item
        for xml_string in separated_xml(f):
            soup = BeautifulSoup(xml_string,'xml')
            if soup.find('us-patent-grant'):
                print item, i, xml_string[177:204]          
                get_data(soup)
            else:
                print item, i, xml_string[177:204],'***********************************soup not found********************************************'
            i+=1
    print 'DONE!!!'
    


Answer: Here is a [tutorial on multi-
threading](http://www.tutorialspoint.com/python/python_multithreading.htm),
because currently that code will run on 1 thread, 1 core.

Remove all try/except statements and handle the code properly. Exceptions are
expensive.

Run a [profiler](http://docs.python.org/2/library/profile.html) to find the
chokepoints, and multi-thread those or find a way to do them less times.

