Inspect a large dataframe for errors arising during merge/combine in python
Question: I hope this is an appropriate question for here. If not, let me know, and I
will remove it immediately.

Question:

How can I use python to inspect (visually?) a large dataset for errors that
arise during combination?

Background:

I am working with several large (but not, you know "Big") datasets that I
combine to form one larger dataset. This new set is ~2.5G in size, so it does
not fit in most spreadsheet programs, or at least not in the ones I've tried
(MS Excel, OpenOffice).

The process to create the final dataset uses fuzzy matching (via
`fuzzywuzzy`), and I want to inspect the results of the matching to see if
there are any errors introduced.

As of now, I have tried importing the entire set into a `pandas` dataframe.
This DF has 64 columns, so when I simply do something like `df.head()` the
resulting displayed info obviously does not show all the columns; I thus ruled
out just iterating through multiple `.head()` calls.

There is a similar question about visualizing specific aspects of a dataframe
[here](http://stackoverflow.com/questions/28813057/inspecting-and-visualizing-
gaps-blanks-and-structure-in-large-dataframes). My question is different, I
think, because I don't need to visualize anything about the underlying
structure or types. I just want to visually inspect areas I suspect might have
errors.


Answer: How about slicing your 10-12 rows and then transposing that you have a 64 rows
x 12 columns dataframe. This should be readable provided you don't have very
large index names.

    
    
    import pandas as pd
    import numpy as np
    
    # Set max number of rows, 64 would be enough here but I'm trying to be safe
    pd.set_option('display.max_rows', 500)
    
    df = pd.DataFrame(np.random.randn(1000,64))
    nstart = 100
    # Slice 12 lines starting at nstart, and transpose that...
    df.iloc[nstart:(nstart+13)].T
    

I'm sparing you the output here, but try running the above code.

