The Parameter Devil — How to set them when no validation against groundtruth is possible Question:

I want to kick up a discussion on how people set algorithmic parameters when no validation against groundtruth is possible (maybe because groundtruth just cannot be obtained or is very hard/tedious to obtain). 

I have read numerous papers and implemented the underlying algorithms wherein --- a set of parameters are said to have been set "empirically" --- and often i found that these are the ones that affect the generality of the algorithm (even though the theory underlying the method is elegant, enticing and sound).

I would appreciate it if you could share your thoughts. And, there is no right or wrong answer for this question. I just want to know, how everybody else deals with this.

Background/Source-of-question:

I am a computer scientist working in the areas of image analysis, computer vision, and machine learning and this question has been on the back of my mind for a while as i have faced this dilemma time and again whenever i design a new algorithm and i found myself spending a considerable amount of time tuning the parameters. 

Also, I think, my question here is more general to any area where in computational algorithms are heavily involved, and i want to invite the thoughts of people from all concerned areas.

I wanted to give you some concrete example, just so it helps you think:

--- Take the case of feature detection (lets say circular blobs or salient points). You run some filters (needs parameters) at different scales (scale parameters) and probably threshold the response (threshold parameter). Its usually not possible to get a groundtruth to validate against and thereby automatically tune your parameters in such scenarios.

--- Take any computational framework which involves a lot of signal processing components. There are always parameters to tune and usually there is no groundtruth and when you tune them subjectively on a small random subset of your dataset you will someday encounter the case to which it doesnt generalize.

This parameter devil is more troublesome when you are setting parameters for some intermediate steps in your algorithm. 

And I often found, it is not possible to cast the problem of finding good values for these parameters as an optimization problem with an objective function of which you can take a derivative and thereby use standard optimization algorithms to find good values. 

Also, in many scenarios exposing these parameters to an end-user is not an option, as we often develop applications/software for non-computational end-users (lets say biologists, doctors) and they usually go clueless when you ask them to tune it unless its very intuitive (like approx object size).

Please share your thoughts.
Assuming that there is a ground truth, (at least theoretically) one of the possible ways to overcome the "tediousness" problem is a "bootstrap" ground truth creation. If you already have a decent algorithm that does the job in about, say 80%-90% of the cases, you can run your algorithm on a large set of instances and ask a user to mark only the mistakes. This approach has its own flaws, such as bias towards your algorithm.

However, there are some cases in which there is no ground truth at all, only different system trade-offs. For instance, an image processing system is required to output a sharp, color-accurate, non-noisy image. Obviously, you can't have them all at the same time. In such case you should use objective metrics that can be calculated on the result of your system. (See Imatest, DXO analyzer for image processing). 

Once you have these, there are methods of multi-objective optimization that can create a mapping from the trade-offs (which are clear to the user) to the intrinsic parameters.

In any case, you should never give the user a parameter that he cannot understand. If all fails, simply hard-code the parameter.