Measuring quality of answers from QnA systems
The ranking of the answers is part of the ML process, i.e. a system should be trained to rank the answers according to their relevance. Heuristic measures such as the ones mentioned in your question may offer decent approximations, but as you noticed they are very limited. 

You may be interested in datasets and methods used in shared tasks about QA, for instance https://mrqa.github.io/shared.