Solutions for big data preprecessing for feeding deep neural network models built with TensorFlow 2.0? Currently I am using Python, Numpy, pandas, scikit-learn to do data preprocessing (LabelEncoder, MinMaxScaler, fillna, etc.), and then feeding the processed data to DNN models built with Tensorflow 2.0. This input pipeline meets my needs when data is small enough to fit a PC's RAM.

Now I have some large datasets, more than 10GB, some are larger. I also plan to deploy the models in a production environment, which means there will be new data coming everyday. For DNN model training there is distributed strategy of tensorflow 2.0. But for data preprocessing obviously I cannot use pandas, scikitlearn on the large datasets with one PC. It seems to me I need to use a for-loop where I repeatedly fetch a small part of the data and use it for training?

I am wondering what do people typically use in either experiment or production environment for big data preprocessing?
Should I use Spark(PySpark) and Tensorflow input pipeline?
If most of your data and machine learning pipeline is in Python, then Dask project for Python is a very good candidate. It allows you to scale (certain types of) data-frame operations to datasets more than memory. The good thing about Dask is that it is also easy to scale down if required. Most of the code stays in Python, you don't pay the serialization overhead (which is paid in PySpark for Python -> JVM).