title: Monitoring the progress of xarrays split apply combine using dask
body: <p>For my thesis I'm trying to calculate the SLOM score (see <a href="https://link.springer.com/article/10.1007/s10115-005-0200-2" rel="nofollow noreferrer">https://link.springer.com/article/10.1007/s10115-005-0200-2</a>).
  This score is purely spatial, and I am trying to calculate it for larger datasets. (upwards of a year).</p>
  <p>So far I created a function that calculates SLOM scores per timestep, which returns an xarray dataarray containing the SLOM values.</p>
  <p>I am trying to calculate the SLOM values for each timestep.</p>
  <p>Currently I am doing this using the groupby: split apply combine strategy (<a href="http://xarray.pydata.org/en/stable/groupby.html" rel="nofollow noreferrer">http://xarray.pydata.org/en/stable/groupby.html</a>)</p>
  <pre><code>grouped_by_time = xrDS.groupby(&quot;time&quot;)
  xrDS[&quot;SLOM&quot;]=grouped_by_time.apply(slom_per_timeslice)
  </code></pre>
  <p>In order to speed up the process I am trying to use the dask functionality built into xarray by loading my data as a daskarray:</p>
  <pre><code>xrDS = xr.open_dataset(data_path+file_name, chunks={&quot;lat&quot;:-1,&quot;lon&quot;:-1,&quot;time&quot;: &quot;auto&quot;})
  </code></pre>
  <p>I think the apply function should work with daskarrays, according to the first sentence: <a href="http://xarray.pydata.org/en/stable/dask.html#using-dask-with-xarray" rel="nofollow noreferrer">http://xarray.pydata.org/en/stable/dask.html#using-dask-with-xarray</a></p>
  <p>Now my question is how do I monitor the progress of the groupedby.apply function?
  I tried using the dask progressbar:</p>
  <pre><code>from dask.diagnostics import ProgressBar
  
  with ProgressBar():
    xrDS[&quot;SLOM&quot;]=grouped_by_time.apply(slom_per_timeslice)
  </code></pre>
  <p>Gives the following output:</p>
  <pre><code>[########################################] | 100% Completed |  0.1s
  [########################################] | 100% Completed |  0.1s
  [########################################] | 100% Completed |  0.1s
  [########################################] | 100% Completed |  0.1s
  [########################################] | 100% Completed |  0.1s
  [########################################] | 100% Completed |  0.1s
  [########################################] | 100% Completed |  0.1s
  [########################################] | 100% Completed |  0.1s
  [########################################] | 100% Completed |  0.1s
  </code></pre>
  <p>hundreds of times.</p>
  <p>So how do I properly see the progress of the overall calculation?</p>
  <p>As a note, I am working small tests on a jupyter notebook on my personal desktop, and performing larger runs on the computer of the research group. I only have ssh access to this machine.
  Both cases are single machine, so I think the dask default multithreaded scheduler should suffice.</p>
  <p>I have looked at the dask dashboard, but how would that work with only ssh access?</p>