Is MLlib compulsory to work with distributed data?
Yes. There are a million ways to engineer a solution.

You can use pandas for data wrangling and distribute this task via parallel python.

I personally use python for distributed - data acquisition, data wrangling, and algorithmic computation. This way i get the capability of Multi-machine, multi CPU, &amp; multi GPU.

If you want to use python all they way, i recommend recording all the different options in terms of libraries, then order them by most active in development releases, or by whatever criterion you value the most.