title: Merge DataFrames and discard duplicates values
body: <p>I'm collecting time-indexed data coming from various files, but sometimes there is some overlapping:</p>
  
  <pre><code>df1 = pd.DataFrame([1, -1, -3], columns=['A'], index=pd.date_range('2000-01-01', periods=3))
  df2 = pd.DataFrame([-3, 10, 1], columns=['A'], index=pd.date_range('2000-01-03', periods=3))
  pd.concat([df1, df2])
  
              A
  2000-01-01  1
  2000-01-02 -1
  2000-01-03 -3
  
               A
  2000-01-03  -3
  2000-01-04  10
  2000-01-05   1
  
               A
  2000-01-01   1
  2000-01-02  -1
  2000-01-03  -3
  2000-01-03  -3
  2000-01-04  10
  2000-01-05   1
  </code></pre>
  
  <p><strong>1) How to clean and remove the duplicate lines ?</strong> (here 2000-01-03)</p>
  
  <p><strong>2) More generally, is there a faster / more clever way with <code>pandas</code> to read and merge multiple csv files</strong> than doing manually:</p>
  
  <pre><code>L=[]
  for f in glob.glob('*.csv'):
      L.append(pd.read_csv(f, ...))
  fulldata = pd.concat(L)                   # this can be time consuming
  fulldata.remove_duplicate_lines()         # this can be time consuming too
  </code></pre>