Universal Psychometrics Tasks: difficulty, composition and decomposition
  This note revisits the concepts of task and difficulty. The notion of
cognitive task and its use for the evaluation of intelligent systems is still
replete with issues. The view of tasks as MDP in the context of reinforcement
learning has been especially useful for the formalisation of learning tasks.
However, this alternate interaction does not accommodate well for some other
tasks that are usual in artificial intelligence and, most especially, in animal
and human evaluation. In particular, we want to have a more general account of
episodes, rewards and responses, and, most especially, the computational
complexity of the algorithm behind an agent solving a task. This is crucial for
the determination of the difficulty of a task as the (logarithm of the) number
of computational steps required to acquire an acceptable policy for the task,
which includes the exploration of policies and their verification. We introduce
a notion of asynchronous-time stochastic tasks. Based on this interpretation,
we can see what task difficulty is, what instance difficulty is (relative to a
task) and also what task compositions and decompositions are.
