title: Pandas dataframe CSV reduce disk size
body: <p>for my university assignment, I have to produce a csv file with all the distances of the airports of the world... the problem is that my csv file weight 151Mb. I want to reduce it as much as i can: This is my csv:</p>
  
  <p><a href="https://i.stack.imgur.com/HwMVe.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HwMVe.png" alt="enter image description here"></a></p>
  
  <p>and this is my code:</p>
  
  <pre class="lang-py prettyprint-override"><code># drop all features we don't need
  for attribute in df:
      if attribute not in ('NAME', 'COUNTRY', 'IATA', 'LAT', 'LNG'):
          df = df.drop(attribute, axis=1)
  
  # create a dictionary of airports, each airport has the following structure:
  # IATA : (NAME, COUNTRY, LAT, LNG)
  airport_dict = {}
  for airport in df.itertuples():
      airport_dict[airport[3]] = (airport[1], airport[2], airport[4], airport[5])
  
  # From tutorial 4 soulution:
  airportcodes=list(airport_dict)
  airportdists=pd.DataFrame()
  for i, airport_code1 in enumerate(airportcodes):
      airport1 = airport_dict[airport_code1]
      dists=[]
      for j, airport_code2 in enumerate(airportcodes):
          if j &gt; i:
              airport2 = airport_dict[airport_code2]
              dists.append(distanceBetweenAirports(airport1[2],airport1[3],airport2[2],airport2[3]))
          else:
          # little edit: no need to calculate the distance twice, all duplicates are set to 0 distance
              dists.append(0)
      airportdists[i]=dists
  airportdists.columns=airportcodes
  airportdists.index=airportcodes
  
  # set all 0 distance values to NaN
  airportdists = airportdists.replace(0, np.nan)
  airportdists.to_csv(r'../Project Data Files-20190322/distances.csv')
  </code></pre>
  
  <p>I also tried re-indexing it before saving:</p>
  
  <pre class="lang-py prettyprint-override"><code># remove all NaN values
  airportdists = airportdists.stack().reset_index()
  airportdists.columns = ['airport1','airport2','distance']
  </code></pre>
  
  <p>but the result is a dataframe with 3 columns and 17 million columns and a disk size of 419Mb... quite not an improvement...</p>
  
  <p>Can you help me shrink the size of my csv? Thank you!</p>