Verify that a CSV file contains a backup job for every prod job This is a list of jobs in autosys. What I have to do is make sure that every job in PROD has a accompanying job in BACKUP. 

The jobs are formatted as CAPSER_JOB_01_PP. I read the list into a while loop, not sure if I should take an -r.

I take off the last two characters of the job name, and then grep twice for each area, if both are successful it prints a yes at beginning of line, otherwise it is a no. 

Frankly I can't find matching documentation for the ${job%??}. I don't know what the two question marks stand for. 

I tried using grep -q but it did not work. the &gt; 2&gt;&amp;1 /dev/null did not work either, I had to reverse it to &gt; /dev/null the redirect 2&gt;&amp;1 (as shown). The script works, it is just a clunky way of getting it done. There has to be a better way. 

#!/bin/bash
#bash, version 3.2.25


IFS=,

while read area job machine script
do
if grep  ${job%??} /home/first_spreadsheet.txt | grep BACKUP &gt;/dev/null 2&gt;&amp;1 &amp;&amp; grep  ${job%??} /home/first_spreadsheet.txt | grep PROD  &gt;/dev/null 2&gt;&amp;1 ; then
echo " YES $area $job $machine $script "
else
echo " NO $area $job $machine $script "
fi
sleep 1
done &lt; /home/first_spreadsheet.txt




cat /home/first_spreadsheet.txt
BACKUP, CAPSER_JOB_01_PP, usa-penguin.com, /bin/bash -lc '/usr/bin/run.sh'
PROD, CAPSER_PROD_JOB_01_PS, usa-penguin.com, /bin/bash -lc '/usr/bin/run.sh'
BACKUP, CAPSER_JOB_02_PP, usa-penguin.com, /bin/bash -lc '$HOME/run/script02'
PROD, CAPSER_PROD_JOB_02_PS, usa-penguin.com, /bin/bash -lc '$HOME/run/script02'
BACKUP, CAPSER_JOB_03_PP, usa-penguin.com, /bin/bash -lc '$HOME/run/script03'
PROD, CAPSER_PROD_JOB_03_PS, usa-penguin.com, /bin/bash -lc '$HOME/run/script03'
BACKUP, CAPSER_JOB_04_PP, usa-penguin.com, /bin/bash -lc '$HOME/run/script04'
PROD, CAPSER_PROD_JOB_04_PS, usa-penguin.com, /bin/bash -lc '$HOME/run/script04'
PROD, CAPSER_PROD_JOB_05_PS, usa-penguin.com, /bin/bash -lc '$HOME/run/script05'
PROD, CAPSER_PROD_JOB_06_PS, usa-penguin.com, /bin/bash -lc '$HOME/run/script06'
BACKUP, CAPSER_JOB_07_PP, usa-penguin.com, /bin/bash -lc '$HOME/run/script07'
PROD, CAPSER_PROD_JOB_07_PS, usa-penguin.com, /bin/bash -lc '$HOME/run/script07'
I take off the last two characters of the job name, and then grep twice for each area, if both are successful it prints a yes at beginning of line, otherwise it is a no.
  
  Frankly I can't find matching documentation for the ${job%??}. I don't know what the two question marks stand for.


${job%??} takes the last two characters off the value of $job.
The ${parameter%word} syntax is explained in man bash in the Parameter Expansion section.
The valid patterns in word are shell glob patterns.
A ? in shell glob patterns means any single character.
So two ? mean any two characters.


  I tried using grep -q but it did not work.


Saying "it did not work" always invites the question "what happened?".
I'd recommend to eliminate the phrase "it did not work" from your vocabulary.
I guess one of two things happened:


The command gave an error, perhaps grep: invalid option -- 'q'


This can happen if the implementation of grep in your system does not have such option.

The command did not give an error, but output was not suppressed.


This should be impossible, so I suspect a mistake on your part. If we had the exact command you executed, and the input and output, we could probably explain the mistake.




  the &gt; 2&gt;&amp;1 /dev/null did not work either, I had to reverse it to &gt; /dev/null the redirect 2&gt;&amp;1 (as shown).


It's a bit difficult to understand this sentence.
In any case, &gt; 2&gt;&amp;1 /dev/null is invalid syntax.
Perhaps you meant that instead of 2&gt;&amp;1 &gt;/dev/null you had to write &gt;/dev/null 2&gt;&amp;1 to get the desired effect.
Yes the meaning of these two ways are different.
The effect of 2&gt;&amp;1 &gt;/dev/null is that stderr will be written to stdout and the original stdout will be discarded.
The effect of &gt;/dev/null 2&gt;&amp;1 is that both stdout and stderr will be discarded.
This works by thinking of pointer assignments,
I can go into more detail if needed.


  The script works, it is just a clunky way of getting it done. There has to be a better way.


A couple of things can be improved:


If the file /home/first_spreadsheet.txt exists, then grep should not write anything to stderr, so no need to redirect that
It would be better to store repeatedly used values like ${job%??} and /home/first_spreadsheet.txt in helper variables
It's good limit the scope of changing the value of IFS to avoid unintended effects later in the script. You can do that by prefixing the command that needs it, in this example read
To avoid redirecting the output of individual grep commands, you could group them together and redirect the output of the grouping


Putting the above together (and then some),
you could write a bit simpler like this:

path=/home/first_spreadsheet.txt
while IFS=, read area job machine script
do
    job0=${job%??}
    if { grep $job0 "$path" | grep BACKUP &amp;&amp; grep $job0 "$path" | grep PROD; } &gt;/dev/null; then
        printf " YES "
    else
        printf " NO "
    fi
    echo "$area $job $machine $script"
done &lt; "$path"