title: How to monitor progress of ssh file transfer and output to a log file
body: <p>I'm writing a bash script to periodically transfer data to a remote system.  I have a local command that generates a stream, and a remote command that consumes it, so I'm doing something like this:</p>
  
  <pre><code>generate_data | ssh remoteserver.example.com consume_data
  </code></pre>
  
  <p>(Where I have ssh keys set up so I can do this non-interactively.)  This works fine.  However, since this will be an automated process (running as a cron job) and may sometimes be transferring large amounts of data on limited bandwidth, I'd like to be able place periodic progress updates in my log file.  I had thought to use <code>pv</code> (pipe viewer) for this, and this is the best I could come up with:</p>
  
  <pre><code>generate_data | pv -fb | ssh remoteserver.example.com consume_data
  </code></pre>
  
  <p>Again, it works... <strong>but</strong> pv was really written with terminal output in mind, so I end up with a mess in the log that looks like</p>
  
  <pre><code>2.06MB^M2.19MB^M2.37MB^M 2.5MB^M2.62MB^M2.87MB^M3MB^M3.12MB^M3.37MB
  </code></pre>
  
  <p>I'd prefer log messages along the lines of</p>
  
  <pre><code>&lt;timestamp&gt; 2.04MB transferred...
  &lt;timestamp&gt; 3.08MB transferred...
  </code></pre>
  
  <p>If anybody has any clever ideas of how to do this, either with different arguments to <code>pv</code> or via some other mechanism, I'd be grateful.</p>
  
  <p><strong>EDIT:</strong> Thanks for the answers so far.  Of course there are a lot of possible home-brewed solutions; I was hoping to find something that would work "out of the box."  (Not that I'm ruling out home-brewed; it may be the easiest thing in the end.  Since <code>pv</code> already does 98% of what I need, I'd prefer not to re-invent it.)</p>
  
  <p><em><strong>PostScript:</em></strong> Here's the line I ended up using, in hopes it might help someone else at some point.</p>
  
  <pre><code>{ generate_data | pv -fbt -i 60 2&gt;&amp;3 | ssh remoteserver consume_data } 3&gt;&amp;1 | awk -vRS='\r' '{print $1 " transferred in " $2; fflush();}' &gt;&gt; logfile
  </code></pre>