Is pyspark streaming suitable for machine learning/ scientific computing?
Machine learning is process of learn from data. First you train your model and then use that on top of data stream.
Data can be processed as mini, micro or even real time, depends on amount of data it is generating in a particular time.
Flume and Kafka are used to fetch data on real time and store on HDFS or can be fed to Spark with Spark streaming pointing to flume sink.
I am new to spark and have to write a streaming application that has to perform tasks like fast fourier transformations and some machine learning stuff like classification/regression with svms etc. I want to do this in pyspark, because of python's huge variety of modules like numpy, scikit-learn etc. My question is, is it possible to do such stuff in a streaming application? As far as I know, spark uses dstreams. Are these streams convertible to something like numpy arrays or anything similar that can serve as an input for python functions?
Thx