title: How to count input and output rows on the Spark SQL API from Java?
body: <p>I am trying to count the number of rows that a Java process reads and writes. The process is using the SQL API dealing with Datasets of Row. Adding <code>.count()</code> at various points seems to slow it down a lot, even if I do a <code>.persist()</code> prior to those points. </p>
  
  <p>I have also seen code that does a </p>
  
  <pre><code>.map(row -&gt; {
    accumulator.add(1);
    return row;
  }, SomeEncoder)
  </code></pre>
  
  <p>which works well enough but the deserialization and re-serialization of the whole row seems unnecessary and it isn't mentally automatic since one has to come up with the correct SomeEncoder at each point.</p>
  
  <p>A third option is maybe to call a UDF0 that does the counting and then drop the dummy object it would return but I'm not sure if Spark would be allowed to optimize the whole code away if it can tell the UDF0 isn't changing the output.</p>
  
  <p>Is there a good way of counting without deserializing the rows? Or alternatively, is there a method that does the equivalent of  Java's streams' <code>.peek()</code> where the returned data isn't important?</p>
  
  <p>EDIT: to clarify, the job isn't <em>just</em> counting. The counting is just for record-keeping purposes. The job is doing other things. In fact, this is a pretty generic problem, I've got lots of jobs that are doing some transformations on data and saving them somewhere, I just want to keep a running record of how many rows these jobs read and wrote.</p>
  
  <p>Thank you</p>