title: How to get the row top 1 in Spark Structured Streaming?
body: <p>I have an issue with Spark Streaming (Spark 2.2.1). I am developing a real time pipeline where first I get data from Kafka, second join the result with another table, then send the Dataframe to a ALS model (Spark ML) and it return a streaming Dataframe with one additional column predit. The problem is when I tried to get the row with the highest score, I couldn't find a way to resolve it.</p>
  
  <p>I tried:</p>
  
  <ol>
  <li>Apply SQL functions like <code>Limit</code>, <code>Take</code>, <code>sort</code></li>
  <li><code>dense_rank()</code> function</li>
  <li>search in StackOverflow</li>
  </ol>
  
  <p>I read <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#operations-on-streaming-dataframesdatasets" rel="nofollow noreferrer">Unsupported Operations</a> but doesn't seem to be much there.</p>
  
  <p>Additional with the highest score I would send to a Kafka queue </p>
  
  <p>My code is as follows:</p>
  
  <pre><code>val result = lines.selectExpr("CAST(value AS STRING)")
  .select(from_json($"value", mySchema).as("data"))
  //.select("data.*")
  .selectExpr("cast(data.largo as int) as largo","cast(data.stock as int) as stock","data.verificavalormax","data.codbc","data.ide","data.timestamp_cli","data.tef_cli","data.nombre","data.descripcion","data.porcentaje","data.fechainicio","data.fechafin","data.descripcioncompleta","data.direccion","data.coordenadax","data.coordenaday","data.razon_social","data.segmento_app","data.categoria","data.subcategoria")
  result.printSchema()
  
  val model = ALSModel.load("ALSParaTiDos")
  
  val fullPredictions = model.transform(result)
  
  //fullPredictions is a streaming dataframe with a extra column "prediction", here i need the code to get the first row
  
  val query = fullPredictions.writeStream.format("console").outputMode(OutputMode.Append()).option("truncate", "false").start()
     query.awaitTermination()
  </code></pre>
  
  <p><strong>Update</strong></p>
  
  <p>Maybe I was not clear, so I'm attaching an image with my problem. Also I wrote a more simple code to complement it: <a href="https://gist.github.com/.../9193c8a983c9007e8a1b6ec280d8df25" rel="nofollow noreferrer">https://gist.github.com/.../9193c8a983c9007e8a1b6ec280d8df25</a>
  detailing what i need. Please I will appreciate any help :)</p>
  
  <p><a href="https://i.stack.imgur.com/7u3iZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7u3iZ.png" alt="enter image description here"></a></p>