title: Remove pandas rows with duplicate indices
body: <p>How to remove rows with duplicate index values?</p>
  <p>In the weather DataFrame below, sometimes a scientist goes back and corrects observations -- not by editing the erroneous rows, but by appending a duplicate row to the end of a file.</p>
  <p>I'm reading some automated weather data from the web (observations occur every 5 minutes, and compiled into monthly files for each weather station.) After parsing a file, the DataFrame looks like:</p>
  <pre><code>                      Sta  Precip1hr  Precip5min  Temp  DewPnt  WindSpd  WindDir  AtmPress
  Date                                                                                      
  2001-01-01 00:00:00  KPDX          0           0     4       3        0        0     30.31
  2001-01-01 00:05:00  KPDX          0           0     4       3        0        0     30.30
  2001-01-01 00:10:00  KPDX          0           0     4       3        4       80     30.30
  2001-01-01 00:15:00  KPDX          0           0     3       2        5       90     30.30
  2001-01-01 00:20:00  KPDX          0           0     3       2       10      110     30.28
  </code></pre>
  <p>Example of a duplicate case:</p>
  <pre><code>import pandas 
  import datetime
  
  startdate = datetime.datetime(2001, 1, 1, 0, 0)
  enddate = datetime.datetime(2001, 1, 1, 5, 0)
  index = pandas.DatetimeIndex(start=startdate, end=enddate, freq='H')
  data1 = {'A' : range(6), 'B' : range(6)}
  data2 = {'A' : [20, -30, 40], 'B' : [-50, 60, -70]}
  df1 = pandas.DataFrame(data=data1, index=index)
  df2 = pandas.DataFrame(data=data2, index=index[:3])
  df3 = df2.append(df1)
  
  df3
                         A   B
  2001-01-01 00:00:00   20 -50
  2001-01-01 01:00:00  -30  60
  2001-01-01 02:00:00   40 -70
  2001-01-01 03:00:00    3   3
  2001-01-01 04:00:00    4   4
  2001-01-01 05:00:00    5   5
  2001-01-01 00:00:00    0   0
  2001-01-01 01:00:00    1   1
  2001-01-01 02:00:00    2   2
  </code></pre>
  <p>And so I need <code>df3</code> to eventually become:</p>
  <pre><code>                       A   B
  2001-01-01 00:00:00    0   0
  2001-01-01 01:00:00    1   1
  2001-01-01 02:00:00    2   2
  2001-01-01 03:00:00    3   3
  2001-01-01 04:00:00    4   4
  2001-01-01 05:00:00    5   5
  </code></pre>
  <p>I thought that adding a column of row numbers (<code>df3['rownum'] = range(df3.shape[0])</code>) would help me select the bottom-most row for any value of the <code>DatetimeIndex</code>, but I am stuck on figuring out the <code>group_by</code> or <code>pivot</code> (or ???) statements to make that work.</p>