Multiple file reading performance issues
from mailing list

```
Hi,

I'm using pandas to read in a set of tab delimited files into DataFrames. Each file has only 3-5 columns and roughly 20,000 rows. The files have a column that I'd like to index by, but some of the entries are duplicates. I find that it takes roughly 5 seconds to read a set of three such files into memory, drop duplicates, and index using "read_table". I.e. to do:

dict_of_dfs = {}
for filename in three_filenames:
  df = pandas.read_table(filename, sep="\t")
  # drop duplicates: use "entry_id" field
  df = df.drop_duplicates("entry_id")
  df = df.set_index("entry_id")
  dict_of_dfs[filename] = df

So to do the above loop body three times takes 5 seconds, which seems slow to me, unless I am missing something. I'm wondering if this is the expected performance for dataframes of this size, or if I'm misusing something. It seems like IO should not be a bottleneck with files that are this small (keep in mind the max number of entries is less than 30k).

Any advice on how to optimize this would be greatly appreciated! Thanks.
```
