Python Pandas Compare 2 Large DataFrames of Text for Similarity
Question: I have two large dataframes I want to compare. I want a comparison result
capable of a column and / or row wise comparison of similarities by percent.
_This part is simple._ However, I want to be able to make the comparison
ignore differences based upon value criteria. A small example is below.

    
    
    d1 = {'Sample':pd.Series([101,102,103]), 
          'Col1':pd.Series(['AA','--','BB']), 
          'Col2':pd.Series(['AB','AA','BB'])}
    d2 = {'Sample':pd.Series([101,102,103]), 
          'Col1':pd.Series(['BB','AB','--']),
          'Col2':pd.Series(['AB','AA','AB'])}
    df1 = pd.DataFrame(d1)
    df2 = pd.DataFrame(d2)
    df1 = df1.set_index('Sample')
    df2 = df2.set_index('Sample')
    
    comparison = df1.eq(df2)
    
    # for column stats
    comparison.sum(axis=0) / float(len(df1.index))
    
    # for row stats
    comparison.sum(axis=1) / float(len(df1.columns))
    

My problem is that for when `value1='AA' and value2 = '--'` I want them to be
viewed as equal (so when one is `'--'` basically always be true) but,
otherwise perform a normal Boolean comparison. I need an efficient way to do
this that doesn't include excessive looping as the datasets are quite large.


Answer: Below, I'm interpreting _"when one is '--' basically always be true"_ to mean
that any comparison against `'--'` (no matter what the other value is) should
return True. In that case, you could use

    
    
    mask = (df1=='--') | (df2=='--')
    

to find every location where either `df1` or `df2` is equal to `'--'` and then
use

    
    
    comparison |= mask
    

to update `comparison`. For example,

    
    
    import itertools as IT
    import numpy as np
    import pandas as pd
    np.random.seed(2015)
    
    N = 10000
    df1, df2 = [pd.DataFrame(
        np.random.choice(map(''.join, IT.product(list('ABC'), repeat=2))+['--'], 
                         size=(N, 2)),
        columns=['Col1', 'Col2']) for i in range(2)]
    
    
    comparison = df1.eq(df2)    
    mask = (df1=='--') | (df2=='--')
    comparison |= mask
    
    # for column stats
    column_stats = comparison.sum(axis=0) / float(len(df1.index))
    
    # for row stats
    row_stats = comparison.sum(axis=1) / float(len(df1.columns))
    

