title: Calculate Word occurrences from file in bash
body: <p>I'm sorry for the very noob question, but I'm kind of new to <code>bash</code> programming (started a few days ago). Basically what I want to do is keep one file with all the word occurrences of another file</p>
  
  <p>I know I can do this:</p>
  
  <pre><code>sort | uniq -c | sort
  </code></pre>
  
  <p>the thing is that after that I want to take a second file, calculate the occurrences again and update the first one. After I take a third file and so on.</p>
  
  <p>What I'm doing at the moment works without any problem (I'm using <code>grep</code>, <code>sed</code> and <code>awk</code>), but it looks pretty slow.</p>
  
  <p>I'm pretty sure there is a very efficient way just with a command or so, using <code>uniq</code>, but I can't figure out.</p>
  
  <p>Could you please lead me to the right way?</p>
  
  <p>I'm also pasting the code I wrote:</p>
  
  <pre><code>#!/bin/bash
  #   count the number of word occurrences from a file and writes to another file #
  #   the words are listed from the most frequent to the less one                 #
  
  touch .check                # used to check the occurrances. Temporary file
  touch distribution.txt      # final file with all the occurrences calculated
  
  page=$1             # contains the file I'm calculating
  occurrences=$2          # temporary file for the occurrences
  
  # takes all the words from the file $page and orders them by occurrences
  cat $page | tr -cs A-Za-z\' '\n'| tr A-Z a-z &gt; .check
  
  # loop to update the old file with the new information
  # basically what I do is check word by word and add them to the old file as an update
  cat .check | while read words
  do
      word=${words}       # word I'm calculating
      strlen=${#word}     # word's length
      # I use a black list to not calculate banned words (for example very small ones or inunfluent words, like articles and prepositions
      if ! grep -Fxq $word .blacklist &amp;&amp; [ $strlen -gt 2 ]
      then
          # if the word was never found before it writes it with 1 occurrence
          if [ `egrep -c -i "^$word: " $occurrences` -eq 0 ]
          then
              echo "$word: 1" | cat &gt;&gt; $occurrences
          # else it calculates the occurrences
          else
              old=`awk -v words=$word -F": " '$1==words { print $2 }' $occurrences`
              let "new=old+1"
              sed -i "s/^$word: $old$/$word: $new/g" $occurrences
          fi
      fi
  done
  
  rm .check
  
  # finally it orders the words
  awk -F": " '{print $2" "$1}' $occurrences | sort -rn | awk -F" " '{print $2": "$1}' &gt; distribution.txt
  </code></pre>