title: updating a pickle file from a dataframe values
body: <p>i am having a huge pickle file which needs to be updated in every 3 hrs from a dailydata file(a csv file.)</p>
  
  <p>there are two field named TRX_DATE and TIME_STAMP in each two having values like 24/11/2015  and 24/11/2015 10:19:02 respectively.(also 50 additionl fields are there)</p>
  
  <p>so what i am doing is first reading the huge pickle to a dataframe. Then dropping any values for today's date by comparing with TRX_DATE field.</p>
  
  <p>Then reading that csv file to another dataframe. then appending both dataframe and again creating new pickle.</p>
  
  <p>my scripts looks like </p>
  
  <pre><code>import pandas as pd
  import datetime as dt
  import pickle
  
  df = pd.read_pickle('hugedata pickle')
  Today = dt.datetime.today()
  df = df[(df.TRX_DATE &gt; Today)]  #delete any entries for today in main pickle
  
  df1 = pd.read_csv(daily data csv file)
  
  df = df.append(df1,ignore_index=True)
  
  df.to_pickle('same huge data pickle')
  </code></pre>
  
  <p>problem is as follows</p>
  
  <p>1.it is taking huge memory as well as time reading that huge pickle.</p>
  
  <p>2.i need to append df1 to df and only columns from df should only remain and it should exclude if any new column from df1 getting appended. But i am getting new column values having NUN values at so many places.</p>
  
  <p>So need assistance on these things</p>
  
  <p>1.is there way that i will read the small sized csv only and append to pickle file ...(or reading that pickle is mandatory)</p>
  
  <p>2.can it be done like converting the csv to pickle and merge two pickles. by load ,dump method (actually never used that)</p>
  
  <p>3.how to read time  from TIME_STAMP field  and getting datas between two timestamp (filtering by TIME_STAMP).and upadting that to main pickle.previously i am filtering by TRX_DATE values.</p>
  
  <p>Is there a better way--- please suggest.</p>