title: Bash help tallying/parsing substrings
body: <p>I have a shell script I wrote a while back, that reads a word list (HITLIST), and recursively searches a directory for all occurrences of those words. Each line containing a "hit" is appended to file (HITOUTPUT). </p>
  
  <p>I have used this script a couple of times over the last year or so, and have noticed that we often get hits from frequent offenders, and that it would be nice if we kept a count of each "super-string" that is triggered, and automatically remove repeat offenders. </p>
  
  <p>For instance, if my word list contains "for" I might get a hundred hits or so for "foreign" or "form" or "force". Instead of validating each of these lines, it would be nice to simply wipe them all with one "yes/no" dialog per super-string.</p>
  
  <p>I was thinking the best way to do this would be to start with a word from the hitlist, and record each unique occurrence of the super-string for that word (go until you are book-ended by why space) and go from there. </p>
  
  <p>So on to the questions ...</p>
  
  <ol>
  <li>What would be a good and efficient way to do this? My current idea
  was to read in the file as a string, perform my counts, remove
  repeat offenders from the the file input string, and output, but this is 
  proving to be a little more painful that I first suspected. </li>
  <li>Would any specific data type/structure be preferred for this type of
  work?</li>
  <li>I have also thought about building the super-string count as I
  create the HitOutput file, but I could not figure out a clean way of
  doing this either. Any thoughts or suggestions?</li>
  </ol>
  
  <p>A sample of the file I am reading in, and my code for reading in and traversing the hitlist and creating the HitOutput file below:     </p>
  
  <pre><code># Loop through hitlist list
  
      while read -re hitlist || [[ -n "$hitlist" ]]
      do
  
          # If first character is "#" it's a comment, or line is blank, skip
          if [ "$(echo $hitlist | head -c 1)" != "#" ]; then
  
              if [ ! -z "$hitlist" -a "$histlist" != "" ]; then
  
                  # Parse comma delimited hitlist
                  IFS=',' read -ra categoryWords &lt;&lt;&lt; "$hitlist"
  
                  # Search for occurrences/hits for each hit
                  for categoryWord in "${categoryWords[@]}"; do
                      # Append results to hit output string
                      eval 'find "$DIR" -type f -print0 | xargs -0 grep -HniI "$categoryWord"' &gt;&gt; HITOUTPUT
                  done
  
              fi
          fi
  done &lt; "$HITLIST"
  </code></pre>
  
  <blockquote>
    <p>src/fakescript.sh:1:Never going to win the war you mother!</p>
    
    <p>src/open_source_licenses.txt:6147:May you share freely, never taking more than you give.</p>
    
    <p>src/open_source_licenses.txt:8764:May you share freely, never taking more than you give.</p>
    
    <p>src/open_source_licenses.txt:21711:No Third Party Beneficiaries. You agree that, except as otherwise expressly provided in this TOS, there shall be no third party beneficiaries to this Agreement. Waiver and Severability of Terms. The failure of UBM LLC to exercise or enforce any right or provision of the TOS shall
    not constitute a waiver of such right or provision. If any provision of the TOS is found by a court of competent jurisdiction to be invalid, the parties nevertheless agree that the court should endeavor to give effect to the parties' intentions as reflected in the provision, and the other provisions of the TOS remain in full force and effect.</p>
    
    <p>src/fakescript.sh:1:Never going to win the war you mother!</p>
  </blockquote>
  
  <p>An example of my hitlist file:</p>
  
  <pre><code># Comment out any category word lines that you do not want processed (the comma delimited lines)
  # -----------------
  
  # MEH
  never,going,to give,you up
  # ----------------
  
  # blah
  word to,your,mother
  </code></pre>