Is pyspark streaming suitable for machine learning/ scientific computing?
Pyspark is used to run program/code/algorithm in spark, which are coded in python language.
For machine leaning, spake has MLlib library packages.
For streaming purpose, spark has Spark streaming lib packages
You can explore Storm as well for real time streaming.
I am new to spark and have to write a streaming application that has to perform tasks like fast fourier transformations and some machine learning stuff like classification/regression with svms etc. I want to do this in pyspark, because of python's huge variety of modules like numpy, scikit-learn etc. My question is, is it possible to do such stuff in a streaming application? As far as I know, spark uses dstreams. Are these streams convertible to something like numpy arrays or anything similar that can serve as an input for python functions?
Thx