title: Memory allocation error on worker 0: std::bad_alloc: CUDA error
body: <p><strong>DESCRIPTION</strong></p>
  <ul>
  <li>I am just trying to gave a trainign and a test set for the model but I get the following errors</li>
  <li>1st data package - <code>train_data = xgboost.DMatrix(data=X_train, label=y_train) </code> Up until I run just this and do training and anything with, only this does not gives an error message</li>
  <li>2nd data package - <code>test_data = xgboost.DMatrix(data=X_test, label=y_test) </code> couple cells down the line, they are not executed together</li>
  </ul>
  <p><strong>ENVIRONMENT</strong></p>
  <ul>
  <li>followed guide - <a href="https://github.com/rapidsai-community/notebooks-contrib/blob/branch-0.14/intermediate_notebooks/E2E/synthetic_3D/rapids_ml_workflow_demo.ipynb" rel="nofollow noreferrer">https://github.com/rapidsai-community/notebooks-contrib/blob/branch-0.14/intermediate_notebooks/E2E/synthetic_3D/rapids_ml_workflow_demo.ipynb</a></li>
  <li><code>conda create -n rapids-0.16 -c rapidsai -c nvidia -c conda-forge -c defaults rapids=0.16 python=3.7 cudatoolkit=10.2</code></li>
  <li>AWS EC2: Deep Learning AMI (Ubuntu 18.04) Version 36.0 - ami-063585f0e06d22308: MXNet-1.7.0, TensorFlow-2.3.1, 2.1.0 &amp; 1.15.3, PyTorch-1.4.0 &amp; 1.7.0, Neuron, &amp; others. NVIDIA CUDA, cuDNN, NCCL, Intel MKL-DNN, Docker, NVIDIA-Docker &amp; EFA support. For fully managed experience, check: <a href="https://aws.amazon.com/sagemaker" rel="nofollow noreferrer">https://aws.amazon.com/sagemaker</a></li>
  <li>AWS EC2 instance - g4dn.4xlarge - 16GB VRAM, 64 GB RAM</li>
  </ul>
  <p><strong>Side Note</strong></p>
  <ul>
  <li>ERROR GB VRAM sizes are NOT 30GB or 15GB
  <ul>
  <li>1 539 047 424 = 1.5 GB,</li>
  <li>3 091 258 960 = 3 GB,</li>
  <li>3 015 442 432 = 3GB,</li>
  <li>3 091 258 960 = 3 GB.</li>
  <li>The GPU has 16 GB VRAM, so I don't think that this answers the question.</li>
  </ul>
  </li>
  </ul>
  <p><strong>ERROR</strong></p>
  <pre><code>---------------------------------------------------------------------------
  XGBoostError                              Traceback (most recent call last)
  &lt;ipython-input-25-7bd66d4fabf4&gt; in &lt;module&gt;
        1 #train = xgboost.DMatrix(data=X, label=y) #ORIGINAL
  ----&gt; 2 test_data = xgboost.DMatrix(data=X_test, label=y_test)
  
  ~/anaconda3/envs/rapids/lib/python3.7/site-packages/xgboost/core.py in __init__(self, data, label, weight, base_margin, missing, silent, feature_names, feature_types, nthread, enable_categorical)
      448             feature_names=feature_names,
      449             feature_types=feature_types,
  --&gt; 450             enable_categorical=enable_categorical)
      451         assert handle is not None
      452         self.handle = handle
  
  ~/anaconda3/envs/rapids/lib/python3.7/site-packages/xgboost/data.py in dispatch_data_backend(data, missing, threads, feature_names, feature_types, enable_categorical)
      543     if _is_cudf_df(data):
      544         return _from_cudf_df(data, missing, threads, feature_names,
  --&gt; 545                              feature_types)
      546     if _is_cudf_ser(data):
      547         return _from_cudf_df(data, missing, threads, feature_names,
  
  ~/anaconda3/envs/rapids/lib/python3.7/site-packages/xgboost/data.py in _from_cudf_df(data, missing, nthread, feature_names, feature_types)
      400             ctypes.c_float(missing),
      401             ctypes.c_int(nthread),
  --&gt; 402             ctypes.byref(handle)))
      403     return handle, feature_names, feature_types
      404 
  
  ~/anaconda3/envs/rapids/lib/python3.7/site-packages/xgboost/core.py in _check_call(ret)
      184     &quot;&quot;&quot;
      185     if ret != 0:
  --&gt; 186         raise XGBoostError(py_str(_LIB.XGBGetLastError()))
      187 
      188 
  
  XGBoostError: [12:32:18] /opt/conda/envs/rapids/conda-bld/xgboost_1603491651651/work/src/c_api/../data/../common/device_helpers.cuh:400: Memory allocation error on worker 0: std::bad_alloc: CUDA error at: ../include/rmm/mr/device/cuda_memory_resource.hpp:68: cudaErrorMemoryAllocation out of memory
  - Free memory: 1539047424
  - Requested memory: 3091258960
  
  Stack trace:
    [bt] (0) /home/ubuntu/anaconda3/envs/rapids/lib/libxgboost.so(+0x13674f) [0x7fad04f7274f]
    [bt] (1) /home/ubuntu/anaconda3/envs/rapids/lib/libxgboost.so(dh::detail::ThrowOOMError(std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, unsigned long)+0x3ad) [0x7fad05190b0d]
    [bt] (2) /home/ubuntu/anaconda3/envs/rapids/lib/libxgboost.so(dh::detail::XGBDefaultDeviceAllocatorImpl&lt;xgboost::Entry&gt;::allocate(unsigned long)+0x1df) [0x7fad051ac11f]
    [bt] (3) /home/ubuntu/anaconda3/envs/rapids/lib/libxgboost.so(thrust::detail::vector_base&lt;xgboost::Entry, dh::detail::XGBDefaultDeviceAllocatorImpl&lt;xgboost::Entry&gt; &gt;::fill_insert(thrust::detail::normal_iterator&lt;thrust::device_ptr&lt;xgboost::Entry&gt; &gt;, unsigned long, xgboost::Entry const&amp;)+0x26d) [0x7fad051d0d0d]
    [bt] (4) /home/ubuntu/anaconda3/envs/rapids/lib/libxgboost.so(xgboost::HostDeviceVector&lt;xgboost::Entry&gt;::Resize(unsigned long, xgboost::Entry)+0xc9) [0x7fad051d1cc9]
    [bt] (5) /home/ubuntu/anaconda3/envs/rapids/lib/libxgboost.so(xgboost::data::SimpleDMatrix::SimpleDMatrix&lt;xgboost::data::CudfAdapter&gt;(xgboost::data::CudfAdapter*, float, int)+0x3df) [0x7fad052259cf]
    [bt] (6) /home/ubuntu/anaconda3/envs/rapids/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create&lt;xgboost::data::CudfAdapter&gt;(xgboost::data::CudfAdapter*, float, int, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, unsigned long)+0x133) [0x7fad051f3aa3]
    [bt] (7) /home/ubuntu/anaconda3/envs/rapids/lib/libxgboost.so(XGDMatrixCreateFromArrayInterfaceColumns+0xc6) [0x7fad0518c286]
    [bt] (8) /home/ubuntu/anaconda3/envs/rapids/lib/python3.7/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7fae60078630]
  </code></pre>
  <p><strong>CODE 2</strong> If I clean a out a restart the notebook that execute them together in 1 cell.</p>
  <pre><code>train_data = xgboost.DMatrix(data=X_train, label=y_train) 
  test_data = xgboost.DMatrix(data=X_test, label=y_test) 
  </code></pre>
  <p><strong>ERROR 2</strong></p>
  <pre><code>---------------------------------------------------------------------------
  XGBoostError                              Traceback (most recent call last)
  &lt;ipython-input-20-f0c3710678a8&gt; in &lt;module&gt;
        1 #train = xgboost.DMatrix(data=X, label=y) #ORIGINAL
        2 train_data = xgboost.DMatrix(data=X_train, label=y_train)
  ----&gt; 3 test_data = xgboost.DMatrix(data=X_test, label=y_test)
  
  ~/anaconda3/envs/rapids/lib/python3.7/site-packages/xgboost/core.py in __init__(self, data, label, weight, base_margin, missing, silent, feature_names, feature_types, nthread, enable_categorical)
      448             feature_names=feature_names,
      449             feature_types=feature_types,
  --&gt; 450             enable_categorical=enable_categorical)
      451         assert handle is not None
      452         self.handle = handle
  
  ~/anaconda3/envs/rapids/lib/python3.7/site-packages/xgboost/data.py in dispatch_data_backend(data, missing, threads, feature_names, feature_types, enable_categorical)
      543     if _is_cudf_df(data):
      544         return _from_cudf_df(data, missing, threads, feature_names,
  --&gt; 545                              feature_types)
      546     if _is_cudf_ser(data):
      547         return _from_cudf_df(data, missing, threads, feature_names,
  
  ~/anaconda3/envs/rapids/lib/python3.7/site-packages/xgboost/data.py in _from_cudf_df(data, missing, nthread, feature_names, feature_types)
      400             ctypes.c_float(missing),
      401             ctypes.c_int(nthread),
  --&gt; 402             ctypes.byref(handle)))
      403     return handle, feature_names, feature_types
      404 
  
  ~/anaconda3/envs/rapids/lib/python3.7/site-packages/xgboost/core.py in _check_call(ret)
      184     &quot;&quot;&quot;
      185     if ret != 0:
  --&gt; 186         raise XGBoostError(py_str(_LIB.XGBGetLastError()))
      187 
      188 
  
  XGBoostError: [15:20:36] /opt/conda/envs/rapids/conda-bld/xgboost_1603491651651/work/src/c_api/../data/../common/device_helpers.cuh:400: Memory allocation error on worker 0: std::bad_alloc: CUDA error at: ../include/rmm/mr/device/cuda_memory_resource.hpp:68: cudaErrorMemoryAllocation out of memory
  - Free memory: 3015442432
  - Requested memory: 3091258960
  
  Stack trace:
    [bt] (0) /home/ubuntu/anaconda3/envs/rapids/lib/libxgboost.so(+0x13674f) [0x7f7eea73674f]
    [bt] (1) /home/ubuntu/anaconda3/envs/rapids/lib/libxgboost.so(dh::detail::ThrowOOMError(std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, unsigned long)+0x3ad) [0x7f7eea954b0d]
    [bt] (2) /home/ubuntu/anaconda3/envs/rapids/lib/libxgboost.so(dh::detail::XGBDefaultDeviceAllocatorImpl&lt;xgboost::Entry&gt;::allocate(unsigned long)+0x1df) [0x7f7eea97011f]
    [bt] (3) /home/ubuntu/anaconda3/envs/rapids/lib/libxgboost.so(thrust::detail::vector_base&lt;xgboost::Entry, dh::detail::XGBDefaultDeviceAllocatorImpl&lt;xgboost::Entry&gt; &gt;::fill_insert(thrust::detail::normal_iterator&lt;thrust::device_ptr&lt;xgboost::Entry&gt; &gt;, unsigned long, xgboost::Entry const&amp;)+0x26d) [0x7f7eea994d0d]
    [bt] (4) /home/ubuntu/anaconda3/envs/rapids/lib/libxgboost.so(xgboost::HostDeviceVector&lt;xgboost::Entry&gt;::Resize(unsigned long, xgboost::Entry)+0xc9) [0x7f7eea995cc9]
    [bt] (5) /home/ubuntu/anaconda3/envs/rapids/lib/libxgboost.so(xgboost::data::SimpleDMatrix::SimpleDMatrix&lt;xgboost::data::CudfAdapter&gt;(xgboost::data::CudfAdapter*, float, int)+0x3df) [0x7f7eea9e99cf]
    [bt] (6) /home/ubuntu/anaconda3/envs/rapids/lib/libxgboost.so(xgboost::DMatrix* xgboost::DMatrix::Create&lt;xgboost::data::CudfAdapter&gt;(xgboost::data::CudfAdapter*, float, int, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, unsigned long)+0x133) [0x7f7eea9b7aa3]
    [bt] (7) /home/ubuntu/anaconda3/envs/rapids/lib/libxgboost.so(XGDMatrixCreateFromArrayInterfaceColumns+0xc6) [0x7f7eea950286]
    [bt] (8) /home/ubuntu/anaconda3/envs/rapids/lib/python3.7/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7f8044f8d630]
  </code></pre>