title: Huge latency in spark streaming job
body: <p>I have a near real time spark streaming application for image recognition where receiver gets the input frames from kafka. I have 6 receivers per executor, 5 executors in total, I can see 30 active tasks per iteration on Spark UI.</p>
  
  <p>My problem is spark able to read 850 frames/sec from kafka but processes task very slowly, which is why i am facing backpressure related issues. Within each batch, the task is expected to run few tensorflow models by first loading them using keras.model_loads and then performs other related processing in order to get the prediction from the model. The output of 1st tensorflow model is the input to 2nd tensorflow model which in turns also load another model and perform prediction on top of it. Now finally output of #2 is the input to model #3 which do the same thing, load the model and perform prediction. The final prediction is send back to kafka to another topic. Having this process flow for each task, overall latency to process a single task is coming somewhere between 10 to 15 seconds which is huge for a spark streaming application </p>
  
  <p>Can anyone help me, how can I make this program fast? </p>
  
  <p>Remember I have to use these custom tensorflow models in my program to get the final output.</p>
  
  <p>I have the following thoughts in my mind:</p>
  
  <ul>
  <li><p>Option 1 - Replace spark streaming with structured streaming</p></li>
  <li><p>Option 2 - Break sequential processing and put each sub process in separate RDD i.e. model #1 processing in RDD1, model #2 processing in RDD2 and so on</p></li>
  <li><p>Option 3 - Rewrite custom tensorflow functionality in spark only, currently that is a single python program which I am using with each task. However I am not so sure about this option yet and not even check the feasibility so far. But what I am assuming that if I am able to do that i will have full control over the distribution of models. Therefore may get the fast processing of these task on GPU machines on AWS cluster which is not happening currently.</p></li>
  </ul>