Recently, I was backing up a directory tree using cp -r, when I ran out of space in the receiving drive. I had to carry on with the backup, but to a different destination. Normally, to resume a cp command, you would ask cp to only copy the file if it isn't in the destination. You can see the problem here.Here was my solution:Screw up.# cp -rv sourcedir destdirerror: no space left on device.Make a list of all files that had been copied successfully.# cd destdir# find . >/tmp/alreadycopiedWrite a script that could take any list A and a blacklist B and return a new list C = B \ A, containing every element in A that is not in B. I called it setminus. ***generate list A*** | setminus listB returns C to stdout.Use find and setminus to copy all remaining files to the new destination.# cd sourcedir# find . -type f | setminus /tmp/alreadycopied | xargs -d '\n' -I file cp -v --parents file overflowdestdirIt worked, but I figure this set minus of lists is a common enough problem that the standard UNIX tools must somehow cover this use case, making my script unnecessary. Have any of you run into this problem, and if so, how did you solve it?The setminus script:#!/usr/bin/env python3# ***generate list*** | setminus blacklist# Performs a set minus on its inputs and returns the result. Specifically,# filters a newline-separated list in stdin using blacklist---also a# newline-separated list. If an element of stdin is found in blacklist, it is# excluded from the output. Otherwise, the element is returned to stdout. Very# useful in conjunction with find commands.from sys import *try:    blacklistfile = argv[1]except IndexError:    stderr.write('usage: ***generate list*** | setminus blacklist.\n')    exit(1)# A dict is used instead of a list to speed up searching. This blacklist could potentially be very large!blacklist = {}for line in open(blacklistfile):    blacklist[line] = Truefor line in stdin:    inblacklist = False    try:        inblacklist = blacklist[line]    except KeyError:        pass    if not inblacklist:        stdout.write(line)
Set minus of two newline-terminated lists / generic blacklisting using common household items
command line;find;backup;cp
If your lists are sorted you could use comm -23 to get the unique elements of the first list.  If they are not, you could use grep likefind -type f | grep -vFxf /tmp/alreadyCopied-v will find all the lines without a match-F tells it to use the strings as fixed strings, not as patterns-x matches the whole line instead of the string anywhere in the line-f /tmp/alreadyCopied read the lines to match from the given fileYou'll have to make sure the paths match though, so if find is producing ./dir1/file1 that needs to be the same string in /tmp/alreadyCopiedDo note though, that this general approach will have problems if, say, you have a filename with \n in it.  You could probably redo the whole thing within find with something likefind . -type f -exec test ! -f destdir/{} \; -exec cp -v --parents {} overflowdestdir \;