The Parameter Devil — How to set them when no validation against groundtruth is possible
Assuming that there is a ground truth, (at least theoretically) one of the possible ways to overcome the "tediousness" problem is a "bootstrap" ground truth creation. If you already have a decent algorithm that does the job in about, say 80%-90% of the cases, you can run your algorithm on a large set of instances and ask a user to mark only the mistakes. This approach has its own flaws, such as bias towards your algorithm.

However, there are some cases in which there is no ground truth at all, only different system trade-offs. For instance, an image processing system is required to output a sharp, color-accurate, non-noisy image. Obviously, you can't have them all at the same time. In such case you should use objective metrics that can be calculated on the result of your system. (See Imatest, DXO analyzer for image processing). 

Once you have these, there are methods of multi-objective optimization that can create a mapping from the trade-offs (which are clear to the user) to the intrinsic parameters.

In any case, you should never give the user a parameter that he cannot understand. If all fails, simply hard-code the parameter.