Create new performance indicators (error metrics)
In research, new evaluation measures are proposed all the time but very often for specific use cases: either because it's a new task for which there is no established evaluation standard, or because the new measure provably solves some issues/limitations with the previous evaluation standard.
The purpose of an evaluation measure (or performance indicator),  is to represent the quality of the predictions by any system for a given task. This implies that:

An evaluation measure is selected specifically for a task. Some measures are very common because many tasks can be framed in a standard way, for example as a classification problem.
The central problem of evaluation is how to define &quot;quality&quot; in a formal way. A performance indicator is always a simplified representation of &quot;quality&quot;, but it must correlate at least approximately with what a human expert of the task would call &quot;quality&quot;.

Naturally the first step for developing a new evaluation measure is to select the target task (or group of tasks). Then one has to demonstrate that the new evaluation measure brings some significant advantage in terms of measuring the quality of this task. Of course this is the hard part, and it always depends specifically on the task.
For example, there have been numerous evaluation measures proposed for Machine Translation (there's even a regular Shared Task devoted to the topic). The way a MT evaluation measure is itself evaluated is against human judgements: if the automatic measure correlates strongly with human judgements about the quality of translation, then it's a good measure.