Grouping of similar looking text I have a data frame which has two columns, "Title" and "Description". The title column has a bunch of titles related to clinical lab tests. Unfortunately, most of the titles are a repeat of the same test but, due to minor changes in the titles, titles are shown as unique.

values = [('Complete blood picture', 'AB'), ('Complete BLOOD test', 'AB'), ('blood glucose', 'AB'), ('COMplete blood Profile', 'AB')]

labels = ['title', 'description']
import pandas as pd
labtest = pd.DataFrame.from_records(values, columns = labels) # Create data frame


This is how the data frame looks like. [The actual dataset has many such titles, this is only for the purpose of this question]

Title                       Description
Complete blood test         AB
COMPLETE Blood test\        AB
Blood glucose               AB
Complete blood picture      AB


And this is what I would the like the data frame to look like:

Title                       Description
Blood test                   AB
Blood test                   AB
Blood test                   AB
Blood test                   AB


I would like to search for the word "blood" in each title and if its true, then change the whole title with "Blood test". Is there a way to do this?
One possible solution is the following:

import re
import pandas as pd

pattern = re.compile('blood', re.IGNORECASE)

def change(text):
    if pattern.findall(text):
        return 'Blood test'
    else:
        return text

values = [('Complete blood picture', 'AB'), ('Complete BLOOD test', 'AB'), ('blood glucose', 'AB'), ('COMplete blood Profile', 'AB')]
labels = ['title', 'description']

# Create data frame
labtest = pd.DataFrame.from_records(values, columns=labels)
labtest['title'] = labtest['title'].apply(change)

print labtest


The output was:

        title description
0  Blood test          AB
1  Blood test          AB
2  Blood test          AB
3  Blood test          AB


The first line import the regex (regular expression) module of Python. The line: 

pattern = re.compile('blood', re.IGNORECASE)


creates a regex that finds the word blood ignoring case. The function change, replace the input text with 'Blood test' in case the string 'blood' was found. Finally you used the apply method from pandas DataFrame to transform the column. Finally the apply method, as the name suggests, 'applies' the function change to every value in the 'title' column.  

More info on regular expressions using Python and pandas apply method can be found here and here. If you want to know more about text processing in Python I would recommend you take a look at the pointers in this question.