title: ShellScript: grep+while+cut+awk in a large file = very slow
body: <p>I have this script running in a 1.7GB  text file.</p>
  
  <pre><code>#!/bin/bash
  
  File1=$1.tmp
  File2=$1.modified
  
  grep '^#' $1 &gt; $File2
  grep -v '#' $1 &gt; $File1
  
  while read line; do
          column_four=$(echo $line | cut -d " " -f4)
          final_line=$(echo $line | cut -d " " -f4-5)
          if [ "$column_four" == "0" ]; then
                 beginning_line=$(echo $line | cut -d " " -f1-3)
                 final_line=$(echo $line | cut -d " " -f4-5)
          else
                 final_line=$(echo $line | cut -d " " -f1-2)
          fi
          linef=$(echo "$beginning_line $final_line")
          echo $linef | awk '{printf "%5.0f%12.4f%12.4f%5.0f%12.4f\n", $1, $2, $3, $4, $5}' &gt;&gt; $File2
  done &lt; $File1
  rm -f $File1
  </code></pre>
  
  <p>The problem: it's very, very slow. It creates a new file with columns arranged in a speed of 200KB per minute with a Core2Duo. How can I make it faster?</p>
  
  <p>Thank you.</p>