How do I fix space allocation error on a machine with plenty of space? I'd like to run a model on RStudio Server, but I'm getting this error.


  Error: cannot allocate vector of size 57.8 Gb


This is what my data looks like and it has 10,000 rows.

   latitude longitude                 close_date close_price
1  1.501986  86.35068 2014-08-16 22:25:31.925431   1302246.3
2 36.367095 -98.66428 2014-08-05 06:34:00.165876    147504.5
3 36.599284 -97.92470 2014-08-12 23:48:00.887510    137400.6
4 67.994791  64.68859 2014-08-17 05:27:01.404296    -14112.0


This is my model.

library(caret)
training.samples &lt;- data$close_price %&gt;%
  createDataPartition(p = 0.8, list = FALSE)
train.data  &lt;- data[training.samples, ]
test.data &lt;- datatraining.samples, ]

model &lt;- train(
  close_price~., data = train.data, method = "knn",
  trControl = trainControl("cv", number = 1),
  preProcess = c("center","scale"),
  tuneLength = 1
)


My EC2 instance has more than 57 GB available. This is the memory.

             total       used       free     shared    buffers     cached
Mem:      65951628     830424   65121204         64      23908     215484
-/+ buffers/cache:     591032   65360596
Swap:            0          0          0


And it has enough storage space, too. This is the hard drive space.

Filesystem     1K-blocks    Used Available Use% Mounted on
devtmpfs        32965196      64  32965132   1% /dev
tmpfs           32975812       0  32975812   0% /dev/shm
/dev/xvda1     103079180 6135168  96843764   6% /


And these are details on the machine.

R version 3.5.3 (2019-03-11)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Amazon Linux AMI 2018.03
It doesn't matter that your instance has more than 57.8GB. R is asking for another 57.8GB on top of whatever it is already using. Not to mention any operating system overhead.

Shrink your dataset and see if it what you are doing works at small scale before trying to do it at big scale. Maybe it will work on 9,917 rows and that's good enough. Or it might fail on 500 rows in which case you need to rethink what you are doing.