Stop Spotlight from indexing a drive that is regularly unplugged and reattached
This ended up being a SuperDuper! problem.  It was doing a fresh copy rather than a smart update and when it does that, it sets the drive to be indexed in spotlight rather than retaining your previous setting.  Once I set it to smart update the problem was fixed.