Navigating the jungle of choices for scalable ML deployment I have prototyped a machine learning (ML) model on my local machine and would like to scale it to both train and serve on much larger datasets than could be feasible on a single machine. (The model was built with Python and Keras. It takes in a CSV table of inputs and spits out the corresponding CSV table of predicted outputs.)

My naive "vision" is that I'd have the model reside on a single (master) machine, whereas the data would be equally distributed among several units (whatever unit means: nodes in a "cluster", CPUs, GPUs, ... ?) The model would be projected onto these units and the learned parameters would somehow all synchronize back to the master unit. Similarly, in the case of serving, the same model would be applied to the data that resides on the different units. Does this "vision" sound reasonable? (I have had some experience with parallel computing with MPI and I vaguely remember that's how things work.)

If I were to start from a blank slate, what architecture/infrastructure should I choose to deploy my model in a scalable fashion? Below are some of the confusingly many options I have read about. (I hardly master what each of these things do, so please forgive me if it looks like a laundry list of disparate technologies.)


Kubeflow (i.e., Kubernetes)
PySpark
Amazon SageMaker
Distributed TensorFlow
Dask
Apache Mesos
REST API
Hadoop
... and dozens more


As a pure ML guy (read: Python, Keras, pandas guy coding on a laptop), I'm out of my depth with all the infrastructure jargon that comes with the above links. It's therefore overwhelming to find a starting point, or some kind of "Hello World" example I could relate to. All I want is an architecture from which I can transpose my code and have it run in an efficient and scalable manner. Which one does the job? Despite all the hype around ML, there does not seem to be any comprehensive map or comparative review of all these solutions. I did find some comparisons, for example between Spark and SageMaker, or Spark and Dask, but given my illiteracy in these subjects, they only add to the confusion.
If you're looking for distributed training, you're probably looking for Apache Spark. It's not itself a training library (although it comes with Spark MLlib which implements most common algorithms in a distributed way), but Spark core is something other implementations often build on. 

Pyspark is just the Python API for Spark. Mesos is a cluster manager, like YARN, that Spark can use. Hadoop is for our purposes here a storage system (HDFS) and YARN, so might have the data you're reading. Spark builds on that too.

Kubeflow is, AFAIK, a toolkit for running many relevant data science things on the Kubernetes resource manager. If you're using K8S this will help you get some common training and notebook tools running. I don't know how much it specifically assists distributing training.

Dask is a framework for distributing Python code but I haven't used it; I don't think it's a training library per se. Sagemaker is a hosted notebook and serving tool, mostly. (Plug: if you're interested in Sagemaker, but also using Spark, you'd probably like our Databricks platform even more.)

Many libraries have distributed implementations now. TensorFlow has its own distributed mode. Horovod from Uber also helps distributed TensorFlow in a more efficient way. There's TensorFlowOnSpark too. (Plug again, because this is near and dear to my heart: HorovodEstimator = Horovod + Spark)

MXnet, xgboost also have distributed implementations on Spark. There's BigDL and deeplearning4j, also distributed deep learning for Spark.

And there's more. You're probably going to run into Spark no matter what as its even more commonly used for ETL and other processing related to data science. Then pick your preferred platform. Then pick some distributed training tools to try on your problem.