title: Best practices to prevent duplicate data (pandas/postgres)
body: <p>I am trying to prevent duplicate data from being added to a database based on three columns. In pandas, this is simple because I can do <code>drop_duplicates(subset = ['customer_id', 'call_time', 'employee_id'], inplace = True, keep = 'last']</code></p>
  
  <p>I recently installed postgresql and am loading data with SQLAlchemy with <code>to_sql</code>. </p>
  
  <p>Is it logical to read from my database at the start of the file, find the maximum date that exists in the file, and then only read/modify new data which occurs after that date? Say the max date in the file is 12/04, then I would add files which are 12/05 onward <code>(max_date + pd.Timedelta(days = 1))</code> only to make sure I am not adding duplicate information. </p>
  
  <p>My current setup, since I am only using .csv files, is to always read the full month (current month) of data to be safe. Then I overwrite a month-to-date csv file over and over again every day, adding one new day of data each time.</p>
  
  <pre><code>start_date = (central_time - pd.Timedelta(days = 0))- pd.offsets.MonthBegin()
  end_date = central_time
  days = ((pd.Timestamp(end_date) - pd.Timestamp(start_date)) / np.timedelta64(1, 'D')).astype(int)
  </code></pre>