title: Jupyter ipython kernel dies on large file loading
body: <p>I have a huge binary file of size ~10gbs which I want to load into a pandas dataframe on my Jupyter notebook. I am using the following code for creating the dataframe:</p>
  <pre><code>df = pd.DataFrame(np.fromfile('binary_file.dat', dtype = mydtype)) #the file has over 20 columns of dtype '&lt;f8'
  </code></pre>
  <p>Everytime I run this command, my kernel dies. On debugging, I found that the np.fromfile command goes through but pd.dataframe command is the one which causes the crash. I am running this on a 4 core, 16 GB Ubuntu AWS server. I tried setting</p>
  <blockquote>
  <p>os.environ['KMP_DUPLICATE_LIB_OK'] = 'True'</p>
  </blockquote>
  <p>as per a Stackoverflow answer, but it didn't help. How can I read this file without crashing the kernel? Is it possible to do without increasing the server RAM?</p>
  <p>Any and all assistance is appreciated. Thank you</p>