Data Scientists Shouldn't Work for the Tools
The Modin view of Scaling Pandas - Towards Data Science
Comparing Modin with Dask, Ray, Vaex, and RAPIDSRecently, a blog post was written that compared a variety of tools in a set of head to heads. I wanted to take the opportunity to talk about our vision with Modin and where we’d like to take the field of data science.Modin (https://github.com/modin-project/modin) takes a different view of how systems should be built. Modin is designed around enabling the data scientist to be more productive. When I created Modin as a PhD student in the RISELab at UC Berkeley, I noticed a dangerous trend: data science tools are being built with hardware performance in mind, but becoming more and more complex. This complexity and performance burden is being pushed onto the data scientist. These tools optimize for processor time at the cost of the data scientist’s time.This trend is harmful to the overall field of data science: we don’t want to force every data scientist into becoming a distributed systems expert or require that they understand low-level implementation details to avoid getting punished by performance. Modin is disrupting the data science tooling space by prioritizing the data scientists time over hardware time. To this end, Modin has:No upfront cost to learning a new APIIntegration with the Python ecosystemIntegration with Ray/Dask clusters (Run on/with what you have!)Modin started as a drop-in replacement for Pandas, because that is where we saw the biggest need. Using Modin is as simple as pip install modin[ray] or pip install modin[dask] and then changing the import statement:# import pandas as pdimport modin.pandas as pdModin is evolving into more than just a drop-in replacement for pandas (try latest version 0.7.4), and I will discuss this later in this post. First, I’d like to do a more detailed comparison of Modin vs. other libraries since many comparisons often leave out crucial information.These systems aim to solve different problems, so fair comparisons are challenging. I will try to stick to the qualitative properties to avoid bias.Dask Dataframe vs. ModinDask Dataframe (https://github.com/dask/dask) covers roughly 40% of the Pandas API due to how the underlying system is architected. Dask Dataframe is a row store, like many SQL systems before it. However, dataframes are not SQL Tables. This architecture allows Dask Dataframe to scale extremely well, but prevents it from supporting all Pandas APIs. Dask Dataframe also places requirements on the user to add .compute() or .persist() calls into their data science workload to manage when compute is triggered. Users are also required to specify the number of partitions. Dask is more mature with >5 years since first commit.Modin currently covers >80% of the Pandas API. Modin is architected as a flexible column store, so the partitioning within Modin can be changed based on the operation that needs to be computed. This allows Modin to scale algorithms that strict column or row stores could not. Modin also does not place any computation or partitioning burden onto the user. Optimizations on the user’s query are run under the hood, without input from the user. Modin is still in its infancy at 2 years since first commit.Modin integrates with Dask’s scheduling API, so Modin can run on top of Dask clusters as well. More on this comparison here.Vaex vs. ModinVaex (https://github.com/vaexio/vaex) is a query engine on top of HDF5 and Arrow memory mapped files, so if your data is already in HDF5 format, it is a solid choice. Going from other file formats (CSV, JSON, Parquet) into a Vaex compatible format takes non-trivial compute time and disk space. Vaex can support roughly 35% of the Pandas API, and does not support key Pandas features like the Pandas row Index. Vaex has had nearly 6 years since the first commit.As of writing, Modin relies on Pandas for its HDF5 file reader, and does not support the memory-mapped style of querying that Vaex supports. Modin scales to clusters, while Vaex tries to help users avoid the need for clusters with memory mapped files.RAPIDS cuDF vs. ModincuDF (https://github.com/rapidsai/cudf) is a dataframe library with a Pandas-like API that runs on NVIDIA GPUs. cuDF is bottlenecked by the amount of memory (up to 48GB), and spilling over to host memory comes with a high overhead. The performance you get on the GPU is great as long as the data reasonably fits into the GPU memory. You can use multiple GPUs with cuDF and Dask, and the resulting distributed GPU dataframe will have the same general properties outlined above in the Dask section. cuDF has had 3 years since the first commit.As of writing, Modin does not have GPU support, but Modin’s architecture is layered, so Modin could integrate GPU kernels alongside CPU. Because of the flexibility of the underlying architecture Modin can even integrate hybrid CPU/GPU/*PU kernels. With this GPU support, we will need to ensure it is implemented in a way that allows users to avoid thinking about GPU memory constraints. More on this comparison here.Ray vs. ModinModin is the data processing engine for Ray. Ray is a low-level distributed computing framework for scaling Python code, while Modin has higher level APIs for data manipulation and querying.