keep_in_memory=True does not seem to work
Seems like I just used the old code which did not have `keep_in_memory=True` argument, sorry.

Although i encountered a different problem – at 97% my python process just hung for around 11 minutes with no logs (when running dataset.map without `keep_in_memory=True` over around 3 million of dataset samples)... Can you open a new issue and provide a bit more details ? What kind of map operations did you run ? Hey. I will try to find some free time to describe it.

(can't do it now, cause i need to reproduce it myself to be sure about everything, which requires spinning a new Azuree VM, copying a huge dataset to drive from network disk for a long time etc...) @lhoestq loading dataset like this does not spawn 50 python processes:

```
datasets.load_dataset("/preprocessed_2256k/train", num_proc=50)
```

I have 64 vCPU so i hoped it could speed up the dataset loading...

My dataset onlly has images and metadata.csv with text column alongside image file path column now noticed
```
'Setting num_proc from 50 back to 1 for the train split to disable multiprocessing as it only contains one shard
```

Any way to work around this? @lhoestq thanks, [this helped](https://github.com/huggingface/datasets/blob/9d6d16117a30ba345b0236407975f701c5b288d4/src/datasets/arrow_dataset.py#L1053)


UPD: [Fixed](https://github.com/huggingface/datasets/issues/6568#issuecomment-1880817794) . But a new issue came up :(