Title: Using a subshell for parameter substitution with diff
Tags: bash;shell
Question: I'm writing a shell script, and in an effort to make it shorter and easier to read, I'm trying to use nested subshells to pass parameters to diff.

Here's what I have: 

```if
  diff -iy '$(sort '$(awk 'BEGIN { FS = "|" } ; {print $1}' new-participants-by-state.csv)' '$(awk 'BEGIN { FS = "|" } ; {print $1}' current-participants-by-state.csv)')' &gt; /dev/null;
then  
  echo There is no difference between the files. &gt; ./participants-by-state-results.txt;
else  
  diff -iy '$(sort '$(awk 'BEGIN { FS = "|" } ; {print $1}' new-participants-by-state.csv)' '$(awk 'BEGIN { FS = "|" } ; {print $1}' current-participants-by-state.csv)')' &gt; ./participants-by-state-results.txt;
fi
```

When I run the script, I keep getting ```diff: extra operand 'AL'```

I'd appreciate any insight into why this is failing. I think I'm pretty close. Thanks!

Comment: That's an awful lot for one command line.  Especially since you're having trouble with it, I think you should break it down into smaller chunks and store them in variables.

Comment: I think it's failing because you are trying to use nested subshells to pass parameters to `diff`?  And the lines are so long that the readability has been dramatically reduced.

Here is the accepted answer: Your code is unreadable because the lines are so long:

```if diff -iy '$(sort '$(awk 'BEGIN { FS = "|" } ; {print $1}' new-participants-by-state.csv)' \
       '$(awk 'BEGIN { FS = "|" } ; {print $1}' current-participants-by-state.csv)')' \
       &gt; /dev/null;
then  
    echo There is no difference between the files. &gt; ./participants-by-state-results.txt;
else  
   diff -iy '$(sort '$(awk 'BEGIN { FS = "|" } ; {print $1}' new-participants-by-state.csv)' \
      '$(awk 'BEGIN { FS = "|" } ; {print $1}' current-participants-by-state.csv)')' \
      &gt; ./participants-by-state-results.txt;
fi
```

Repeating whole commands like that is also fairly nasty.  You also have major problems with your use of single quotes; you only have one sort in each set of commands, apparently operating on the combined outputs of two identical ```awk``` commands (whereas you probably need two separate sorts, one for the output of each ```awk``` command); you're not using the ```-F``` option to ```awk``` when you could; you are repeating the gargantuan file names all over the place; and finally, it appears that you are probably wanting to use process substitution, but not actually doing so.

Let's take a step back and formulate the question clearly.


Given two files (```new-participants-by-state.csv``` and ```current-participants-by-state.csv```) find the first pipe-separated field on each line of each file, sort the lists of those fields, and compare the results of the two sorted lists.
If there are no differences, write a message into the output file ```participants-by-state-results.txt```; otherwise, list the differences in the output file.


So, we could use:

```oldfile='current-participants-by-state.csv'
newfile='new-participants-by-state.csv'
outfile='participants-by-state-results.txt'

tmpfile=${TMPDIR:-/tmp}/xx.$$

awk -F'|' '{print $1}' $oldfile | sort &gt; $tmpfile.1
awk -F'|' '{print $1}' $newfile | sort &gt; $tmpfile.2

if diff -iy $tmpfile.1 $tmpfile.2 &gt; $outfile
then echo "There is no difference between the files" &gt; $outfile
fi

rm -f $tmpfile.?
```

If this was going to be the final script, we'd want to put trap handling in place so that the temporary files are not left around unless the script is killed dead with SIGKILL.

However, we can now use process substitution to avoid the temporary files:

```oldfile='current-participants-by-state.csv'
newfile='new-participants-by-state.csv'
outfile='participants-by-state-results.txt'

if diff -iy <(awk -F'|' '{print $1}' $oldfile | sort) \
            <(awk -F'|' '{print $1}' $newfile | sort) &gt; $outfile
then echo "There is no difference between the files" &gt; $outfile
fi
```

Note how the code carefully preserves symmetries where there are symmetries.  Note the use of shortish variable names to avoid the repetition of long file names.  Note that the ```diff``` command is run just once, not twice - throwing away results which are needed later is not very sensible.

You could compress the output I/O redirection even more using:

```{
if diff -iy <(awk -F'|' '{print $1}' $oldfile | sort) \
            <(awk -F'|' '{print $1}' $newfile | sort)
then echo "There is no difference between the files"
fi
} &gt; $outfile
```

That sends the standard output of the enclosed commands to the file.

Of course, CSV might not be the appropriate nomenclature if the files are pipe-separated rather than comma-separated, but that's another matter altogether.

I'm also assuming that the status from ```diff -iy``` works as suggested by the original script; I've not validated that usage of the ```diff``` command.

Comment for this answer: Thanks for such a detailed answer. This is the first time I've tried anything like this, so I appreciate it.

Here is another answer: There are several problems here.

First, you're putting various arguments in single-quotes, which prevents any interpretation being done on them (for example, ```$(....)``` doesn't do anything special inside single-quotes).  You're probably thinking of double-quotes, but those aren't what you want either.

Which brings us to the second problem, that diff and sort expect to be given filenames as arguments, and they operate on the data in those files; you're trying to pass the data directly as arguments, which doesn't work (and I suspect that's the origin of the error you're getting: diff expects exactly two filenames, you're passing more than two participant names, and AL happened to be third on the list and hence the one that diff panicked on).  The usual way to do this is to use intermediate files (and multiple lines in the script), but bash actually has a way of doing this without either of those: process substitution.  Essentially, what it does is run one command with output (or input, but we need output in this case) sent to a named pipe; then it passes the name of the pipe as an argument to another command.  For example, ```diff <(command1) <(command2)``` will give you the differences between the outputs of command1 and command2.  Note that since this is a bash-only feature, you must start the script with ```#!/bin/bash```, not ```#!/bin/sh```.

Third, there's a missing close-parenthesis that makes it a little hard to tell what's supposed to happen.  Are both files supposed to be sorted before the comparison, or only the new-participants file?

Fourth, since the final comparison ignores case (```-i```), you'd better use a case-insensitive sort (```-f```) as well.

Finally, you're doing all of the processing twice if there are any differences.  I'd recommend running the comparison once into a file, then if there were no differences just ignore/overwrite the (empty) file.

Oh, and just a stylistic thing: you don't need semicolons at the end of lines in bash.  You only need semicolons if you're putting more than one command on the same line (and a few other cases like before ```then``` in an ```if``` statement).

Anyway, here's my rewrite:

```#!/bin/bash
if
    diff -iy <(awk 'BEGIN { FS = "|" } ; {print $1}' new-participants-by-state.csv | sort -f) <(awk 'BEGIN { FS = "|" } ; {print $1}' current-participants-by-state.csv | sort -f) &gt;./participants-by-state-results.txt
then
    echo "There is no difference between the files." &gt; ./participants-by-state-results.txt
fi
```

Comment for this answer: Thanks a lot for walking through all the issues and the rewrite!

{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 2, 'CommentCount': 2, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2011:00:11/23/11', 'FavoriteCount': 1.0, 'PostTypeId': 1, 'Tags': '<bash><shell>', 'Title': 'Using a subshell for parameter substitution with diff', 'ViewCount': 1776, 'id': 8236458, 'user_AboutMe': None, 'user_AccountId': 955671.0, 'user_DisplayName': 'user981023', 'user_Reputation': 233.0, 'user_Views': 14.0, 'user_id': 981023.0}}