title: Compute Average response time on hourly basis
body: <p>I am trying to calculate avg response time on hourly basis from the log file which has millions of records, below has the excerpt from log</p>
  
  <p>As of now I am trying with creating temproary file which will have lines with unique id and start time and end time and after that another script will run on this temp file to calcualte avg response time for each hour
  My script takes more than an hour to create temp file.</p>
  
  <p>Is there any way we can do it quicker? or better script whose exceution time is lesser.
  Note:These UNIQID is not coming in sequence.</p>
  
  <pre><code>log file format
  2012-06-04 13:04:19,324 UNIQID1
  2012-06-04 13:04:20,120 UNIQID1
  2012-06-04 13:05:19,324 UNIQID2
  2012-06-04 13:06:20,120 UNIQID2
  2012-06-04 13:07:19,324 UNIQID3
  2012-06-04 13:08:20,120 UNIQID3
  2012-06-04 13:08:49,324 UNIQID4
  2012-06-04 13:09:50,120 UNIQID4
  </code></pre>
  
  <p>Here is my code:</p>
  
  <pre><code>uids=`cat $i|grep "UNIQ" |sort -u` &gt;&gt; $log
  for uid in ${uids}; do  
      count=`grep "$uid" test.log|wc -l`
      if [ "${count}" -ne "0" ]; then
          unique_uids[counter]="$uid"
          let counter=counter+1   
      fi   
  done
  
  
  echo ${unique_uids[@]}   
  echo $counter  
  echo " Unique No:" ${#unique_uids[@]}
  echo uid StartTime EndTime" &gt; $log
  
  for unique_uids in ${unique_uids[@]} ; do
      responseTime=`cat $i|grep "${unique_uids}" |awk '{split($2,Arr,":|,"); print Arr[1]*3600000+Arr[2]*60000+Arr[3]*1000+Arr[4]}'|sort -n`
      echo $unique_uids $responseTime &gt;&gt; $log
  done
  </code></pre>
  
  <p>Thanks for your time!</p>