Bash Script for file auditing, push information to server and be able to view in a web page Edit: The goal of this Question was only seeking code review and guidance in learning or tips for improvement. I apologize if that was not clear or aligned properly with the goals of this forum. 
Script Goal: audit file existence and differences between two red hat Enterprise Linux Servers 


Place information in sql server tables to use for analysis  
Maintain ability to view the diff in a web page 


Process Steps
Set array with Directorys to check 


Set array of file types to look for 
Build a list of existing files for each Environment

Script will solve the following
If the files exist in both servers are they the identical? (md5)* 


If no what is different (Diff)? 
Does the file contain hard coded values (grep) 

View Diff in web page (currently reading text file using into site php) 


Code

DIR=(/Dir/Durp/DurpaDurp/Scriptdir1 /Dir/Durp/DurpaDurp/Scriptdir2 /Dir/Durp/DurpaDurp/Scriptdir3 ) 

f_Type=("*.sh" "*.txt" "*.log")   
f_Type2=("*.img" "*.rpt" )

for((i=0; i&lt;${#DIR[@]}; i++)) 
do
  echo "CHECKING: ${DIR[$i]}"
  cd "${DIR[$i]}" 
      for((x=0; x&lt;${#f_Type[@]}; x++)) 
       do
         echo "FOR FILE TYPE: ${f_Type[$x]}"
        find $PWD -type f -name "${f_Type[$x]}" | sed 's/^/ENV1|/'| column -t &gt;&gt; "$filelistENV1"
      done 
if [[ ${DIR[$i]} == "/Dir/Durp/DurpaDurp/Script3" ]]; 
  then
     for((y=0; y&lt;${#f_Type[@]}; y++)) 
    do
       echo "FOR FILE TYPE: ${f_Type2[$y]}"
      find $PWD -type f -name "${f_Type2[$y]}" | sed 's/^/ENV1|/'| column -t &gt;&gt; "$filelistENV1"
    done 
fi
done 

After these files are output to the text files the script will delete the existing data from a staging table in SQL Server 2008 R2 and insert the new data.

for((i=0; i&lt;${#ENV[@]}; i++))
do
  sqlcmd -S $_DB_CONN -d $_DB  -Q "DELETE FROM ['$_DB']..['$_TABLE'] WHERE ENV = '${ENV[$i]}'";
done

bcp $_DB_CONN.."$_TABLE" in "$filelistENV1" -f "$_SCRIPTDIR/STG.fmt" -e $_ERRDIR/ERROR_STG$(date -d "today" +"%Y%m%d%H%M").txt -S $_DB_CONN -d "$_DB"

the format file creates 2 columns 

AbsoluteFilePath | ENV     
gets a list of files from the database to compare  


`    

sqlcmd -S $_DB_CONN -d $_DB -s "|" -h-1 -m -1 -W -i $_SCRIPTDIR/SQL/EXPORT_COMPARE.sql -o $_INPUT/comp_list.txt  set NOCOUNT ON;



compare the md5sum of the files. 

for i in $(cat "$_INPUT/comp_list.txt")
do 
export filename=$(basename "$i") 
export path=$(dirname "$i") 
env1_md5sum=$(md5sum "$i") 
env1="${env1_md5sum%% *}" 
export tmpdir=("$_TMPDIR$path")

if ssh "$_CONN" stat $path'$filename' \&gt; /dev/null 2\&gt;\&amp;1  
then 
    env2_md5sum=$(ssh $_CONN "cd $path; find -name '$filename' -exec md5sum {} \;")
    env2_md5="${env2_md5sum%% *}"

    if [[ $env1_md5 == $env2_md5 ]]; then 
        echo $filename $path &gt;&gt; "$matchingMD5"
    else 
        echo "md5 does not match, getting copy of file"
        echo "$i" &gt;&gt; "$no_matchMD5"
        mkdir -p $tmpdir
        scp $_CONN:$i $tmpdir
    fi
fi done

run a diff on files that do not match 

for x in $(cat "$no_matchMD5") 
do 
comp_filename=$(basename "$x")
env2file=(/"$ScriptsDir"/tmp"$x")
DIFF=$(diff --ignore-all-space --ignore-blank-lines --brief  "$x" "$env2file" &amp;&gt;/dev/null
Performance pitfalls

Running ssh in a loop tends to be slow. Running it twice for every file in a list is probably extremely slow. There's no easy fix for this. You need to rethink how to solve the problem of matching paths between two systems.

Off the top of my head:


Get the list of paths from both systems, and then try to match those locally. This can be done with one ssh call per system: a major improvement.
For the list of matched paths, get the md5 sums. Again this can be done with one ssh call per system.
Compare the hashes, and build a new list of mismatched files.
For the final comparison of files, you could fetch them one by one to conserve disk space. If the number of remaining files is expected to be small, then one scp call per file might be acceptable. Or if disk space is not an issue, then you could transfer all the files with one call.


A much smaller performance issue is running sed ... | column ... for each file type for each base directory. You could instead make the loop body output only the output of the multiple find calls, and run the sed ... pipeline on the entire loop (writing as done | sed ...).

Looping over arrays

Instead of this:


for((i=0; i&lt;${#DIR[@]}; i++)) 
do
  echo "CHECKING: ${DIR[$i]}"



When you don't need the array indexes, just the elements, you can iterate like this:

for dir in "${DIR[@]}"
do
  echo "CHECKING: $dir"


Most of the loops in the posted script can be replaced with this simpler, more intuitive style.

Simple mistakes


Use . instead of $PWD
Double-quote variables used in command arguments: instead of find $var, write find "$var"
Don't export if you don't need to
Don't create arrays if you need a simple variable: instead of tmpdir=("$_TMPDIR$path") write tmpdir="$_TMPDIR$path"
Strive for simple writing style: instead of env2file=(/"$ScriptsDir"/tmp"$x"), write env2file="$ScriptsDir/tmp$x"