abstract: Utility functions or their equivalents (value functions, objective functions, loss functions, reward functions, preference orderings) are a central tool in most current machine learning systems. These mechanisms for defining goals and guiding optimization run into practical and conceptual difficulty when there are independent, multi-dimensional objectives that need to be pursued simultaneously and cannot be reduced to each other. Ethicists have proved several impossibility theorems that stem from this origin; those results appear to show that there is no way of formally specifying what it means for an outcome to be good for a population without violating strong human ethical intuitions (in such cases, the objective function is a social welfare function). We argue that this is a practical problem for any machine learning system (such as medical decision support systems or autonomous weapons) or rigidly rule-based bureaucracy that will make high stakes decisions about human lives: such systems should not use objective functions in the strict mathematical sense.
  .
  We explore the alternative of using uncertain objectives, represented for instance as partially ordered preferences, or as probability distributions over total orders. We show that previously known impossibility theorems can be transformed into uncertainty theorems in both of those settings, and prove lower bounds on how much uncertainty is implied by the impossibility results. We close by proposing two conjectures about the relationship between uncertainty in objectives and severe unintended consequences from AI systems.
author: Peter Eckersley
Partnership on AI & EFF[^1]
`email@example.com`
title: Impossibility and Uncertainty Theorems
  in AI Value Alignment
  $\:$
  *or* why your AGI should not have a utility function

# Introduction

Conversations about the safety of self-driving cars often turn into discussion of "trolley problems", where the vehicle must make a decision between several differently disastrous outcomes . For the engineers building robots and machine learning systems, however, these questions seem somewhat absurd; the relevant challenges are correctly identifying obstacles, pedestrians, cyclists and other vehicles from noisy and unreliable data sources, and reasonably predicting the consequences of the vehicle's control mechanisms in various conditions. The conceivable circumstances under which a self-driving car's systems might accurately foresee and deliberately act on a genuine "trolley problem" without having been able to avoid that problem in the first place are a tiny portion of possibility space, and one that has arguably received a disproportionate amount of attention.[^2]

There are, however, other kinds of AI systems for which ethical discussions about the value of lives, about who should exist or keep existing, are more genuinely relevant. The clearest category is systems that are *designed* to make decisions affecting the welfare or existence of people in the future, rather than systems that only have to make such decisions in highly atypical scenarios. Examples of such systems include:

-   autonomous or semi-autonomous weapons systems (which may have to make trade-offs between threats to and the lives of friendly and enemy combatants, friendly and enemy civilians, and combatants who have surrendered).[^3]

-   risk assessment algorithms in criminal justice contexts (which at least notionally are making trade-offs between risk to society and the harm that life-shortening incarceration does to individual defendants in the criminal justice system and their families).

-   decision support systems in high-stakes bureucratic settings. For instance a system that helps to allocate scarce resources like doctors' time, laboratory test access and money in a medical system would implicitly or explicitly have to make trade-offs between the life span, quality of life, and fertility of numerous people.

-   autonomous types of open-ended AI taking important actions without their designers' input.

Governmental organizations already have to make decisions that involve weighing benefits to present or future people against risks to present or future people, or trading off benefits (such as wealth) to one group against benefits of another sort (such as freedom) to another. Therefore, we might think that the decision-making tools that they employ could be implemented by any narrow AI system that has to make similar choices. A variety of metrics have been proposed and used to perform cost-benefit analyses in such situations. For example, the US Environmental Protection Agency uses the 'value of a statistical life' (VOSL) metric, which identifies the cost of an intervention that puts people at risk with people's willingness to pay to avoid similar risks (for a critique, see ) or the QALY metric used by the UK's National Health Service, which assigns a given value to each life-year saved or lost in expectation, weighted by the quality of that life. The problem is that existing impossibility theorems apply equally to each of these metrics and therefore to an AI system that implements them.

# Impossibility Theorems in Ethics

## Example: An Impossibility Theorem in Utilitarian Population Ethics

Economists and ethicists study individual utility functions and their aggregations, such as social welfare functions, social choice functions and formal axiologies, and have discovered various impossibility theorems about these functions. Perhaps the most famous of these is Arrow's Impossibility Theorem , which applies to social choice or voting. It shows there is no satisfactory way to compute society's preference ordering via an election in which members of society vote with their individual preference orderings. Fortunately, Arrow's theorem results from some constraints (incomparability of individual's preferences, and the need for incentive compatibility) which may not apply to AI systems.

Unfortunately, ethicists have discovered other situations in which the problem isn't learning and computing the tradeoff between agents' objectives, but that there simply *may not be* such a satisfactory tradeoff at all. The "mere addition paradox" was the first result of this sort, but the literature now has many of these impossibility results. For example, Arrhenius shows that all total orderings of populations must entail one of the following six problematic conclusions, stated informally:

-   **The Repugnant Conclusion** For any population of very happy people, there exists a much larger population with lives barely worth living that is better than this very happy population (this affects the "maximise total wellbeing" objective).

-   **The Sadistic Conclusion** Suppose we start with a population of very happy people. For any proposed addition of a sufficiently large number of people with positive welfare, there is a small number of horribly tortured people that is a preferable addition.[^4]

-   **The Very Anti-Egalitarian Conclusion** For any population of two or more people which has uniform happiness, there exists another population of the same size which has lower total and average happiness, and is less equal, but is better.

-   **Anti-Dominance** Population B can be better than population A even if A is the same size as population B, and every person in A is happier than their equivalent in B.

-   **Anti-Addition** It is sometimes bad to add a group of people B to a population A (where the people in group B are worse of than those in A), but *better* to add a group C that is larger than B, and worse off than B.

-   **Extreme Priority** There is no $n$ such that create of $n$ lives of very high positive welfare is sufficient benefit to compensate for the reduction from very low positive welfare to slightly negative welfare for a single person (informally, "the needs of the few outweigh the needs of the many").

The structure of the impossibility theorem is to show that no objective function or social welfare function can simultaneously satisfy these principles, because they imply a cycle of world states, each of which in turn is required (by one of these principles) to be better than the next. The structure of that proof is shown in Figure .

## Another Impossibility Theorem

Arrhenius's unpublished book contains a collection of many more uncertainty theorems. Here is the simplest, which is a more compelling version of Parfitt's Mere Addition Paradox. It shows the impossibility of an objective function satisfying the following requirements simultaneously:

**The Quality Condition:** There is at least one perfectly equal population with very high welfare which is at least as good as any population with very low positive welfare, other things being equal.

**The Inequality Aversion Condition:** For any triplet of welfare levels A, B, and C, A higher than B, and B higher than C, and for any population A with welfare A, there is a larger population C with welfare C such that a perfectly equal population B of the same size as A$\cup$C and with welfare B is at least as good as A$\cup$C, other things being equal.

**The Egalitarian Dominance Condition:** If population A is a perfectly equal population of the same size as population B, and every person in A has higher welfare than every person in B, then A is better than B, other things being equal.

**The Dominance Addition Condition:** An addition of lives with positive welfare and an increase in the welfare in the rest of the population doesn't make a population worse, other things being equal.

The cyclic structure of the proof is shown in Figure .

## Ethical Impossibility Derives From Competing Objectives

The particular impossibility theorems summarized above, and those in the related literature, result from the incompatibility of several different utilitarian objectives: maximizing total wellbeing, maximizing average wellbeing, and avoiding suffering. Similar problems are also likely to arise when considering almost any kinds of competing objectives, such as attempting to simultaneously maximize different notions of wellbeing, freedom, knowledge, fairness, or other widely accepted or domain-specific ethical goods . We conjecture that the literature contains more impossibility theorems about happiness or wellbeing simply because that objective has been subjected to mathematical modeling and study for well over a hundred years,[^5] while much less effort has gone into the others. Recent work has begun to focus on fairness in decisionmaking contexts in particular, and is already producing its own impossibility results.

# Possible Responses to Impossibility Results

Arrhenius's impossibility results and others like them are quite troubling. They show that we do not presently have a trustworthy framework for making decisions about the welfare or existence of people in the future, and are representative of a broader problem with the inability of the objective functions to reasonably optimise for multiple objectives simultaneously. Next we will consider five possible responses to these impossibility theorems:

## Small-scale evasion:

One response may be to claim that the stakes are simply low for present AI systems that only make small-scale changes to the world. AGI systems, if they exist, may one day need to confront impossibility theorems or difficult conflicting objectives in some cases, but none of that is presently relevant. Unfortunately, it appears that although the stakes are indeed much lower at present, the fundamental tensions that drive these paradoxes are already at play in decisions made by bureaucracies and decision support systems, and small scale decisions can easily violate constraints we would want to see respected. Asking for human feedback helps, but humans often miss important ethical principles, and can easily make chains of decisions that have problematic consequences in proportion to their degree of power or agency. Cases of particular concern are those where ML or algorithmic heuristics are deployed in decision support tools that humans are inadequately inclined to question.

## Value learning:

One might argue that impossibility theorems are only a problem for an AI system if we are attempting to explicitly define an objective function, rather than letting an AI system acquire its objective function in a piecemeal way via a method like co-operative inverse reinforcement learning (CIRL) or other forms of human guidance . Using human feedback to construct objective functions as neural networks appears to be a promising direction,[^6] but if the output of that network is mathematically a utility function or total ordering, these problems will persist at all stages of network training. The theorems do not just reflect a tension between principles that agents are committed to: they are also reflected in the decisions that human supervisors will make when presented with a sequence of pairwise choices.

Therefore, although learning objectives from humans may be a prudent design choice, such learning systems will still need to either violate the ethical intuitions that underpin the various impossibility theorems (some combination of Sections  and ) or explicitly incorporate uncertainty into their outputs (Section ).

## Theory normalization:

One natural response to impossibility theorems is to try harder to figure out good ways of making tradeoffs between the competing objectives that underlie them — in Arrhenius's paradox these are total wellbeing, average wellbeing, and avoidance of suffering. This tradeoff can be simple, such as trying to define a linear exchange rate between those objectives. Unfortunately, linear exchange rates between a total quantity and an average quantity do not make conceptual sense, and it is easy to find cases under which one of them totally outweighs the other.[^7]

One can try more complicated strategies such as trying to write down convex tradeoffs (see eg ) or imagining an explicit "parliament" where different ethical theories and objectives vote against each other, are weighed probabilistically, or are combined using a function like `softmin`[^8]. Unfortunately all of these approaches contain a version of the exchange rate problem, and none of them actually escape the impossibility results: for instance, using any unbounded monotonic non-linear weighting on total utility will eventually lead to the Repugnant Conclusion (*cf* ).

## Accept one of the axioms

Another type of response – one that is commonly pursued in the population ethics literature – is to argue that although each of the axioms above strikes us as undesirable, at least one of them ought to be accepted in some form. For example, it has been argued that although the Repugnant Conclusion might appear undesirable, it is an acceptable consequence of an objective function over populations . We argue that, given the high levels of uncertainty and the lack of political consensus about which of these axioms we ought to accept it would, at present, be irresponsible to explicitly adopt one of the axioms outlined above.

## Treat impossibility results as uncertainty results

The last solution, and the one which we believe may be the most appropriate for deploying AI systems in high-stakes domains, is to add explicit mathematical uncertainty to objective functions. There are at least two ways of doing this: one is to make the objective a mathematical partial order, in which the system can prefer one of two outcomes, believe they are equal, or believe that they are incommensurate. We discuss that approach and demonstrate uncertainty theorems in that formalization in Section . A second approach is to allow objective functions to have some level of confidence or uncertainty about which of two objectives is better (eg, "we're 70% sure that A is better, and 30% sure that B is better"). We discuss that framing and show the existence of formal uncertainty theorems in that framework in Section .

# Uncertainty theorems of the first kind: via partially ordered objective functions

It is already the case that in some problems where humans attempt to provide oversight to AI systems, it is pragmatically better to allow the human to express uncertainty or indifference when asked which of two actions is better. But such systems presently try to construct a totally ordered objective function, and simply interpret these messages from users as containing either no information about the correct ordering (the human doesn't know which is better) or implying that the two choices are close to as good as each other (the human is indifferent). We believe that another type of interpretation is sometimes necessary, which is that the human is *torn* between objectives that fundamentally cannot be traded off against each other.

We can represent this notion with a partially ordered objective function that sometimes says the comparison between world states cannot be known with certainty. Cyclical impossibility theorems like those surveyed in Sections  and can be viewed as evidence of this uncertainty. Here we prove a lower bound on how much uncertainty is evidenced by each such theorem.[^9]

Suppose that $\mathbb{W}$ is the set of possible states of the world, and that there is a an impossibility theorem $T_C$ showing that there is no totally ordered definition of "better" $\leq_Z$ over $\mathbb{W}$ that satisfies a set of constraints $\{C_1, C_2, .. C_n\}$ over the comparison function $Z : \mathbb{W \times W}  \rightarrow \{<, >, =\}$. Each of these constraints are motivated by strong human ethical intuitions, and insist that for a set of pairs of inputs $x$ and $y$, $A(x,y)$ takes some value. We use the notation $x \: \overset{C_i}{\leq_Z} y$ to indicate that $C_i$ requires that $x \leq_Z y$ for some $x$ and $y$). $Z$ also has the usual properties of total orderings:

Antisymmetry: $$a \leq_Z b \land b \leq_Z a \rightarrow a = b$$

Transitivity: $$a \leq_Z b \land b \leq_Z c \rightarrow a \leq_Z c$$

Totality: $$a \leq_Z b \lor b \leq_Z a$$

Suppose further that $T$ is provable by providing a series of example worlds $\{w_1, w_2, ... w_n\}$ such that:

$$w_1 \overset{C_1}{\leq_Z} w_2 \overset{C_2}{\leq_Z}... \overset{C_{n-1}}{\leq_Z} w_n$$ and $$w_n \overset{C_n}{\leq_Z} w_1$$

Violating (by induction) transitivity, $T_C$ shows that no totally ordered $\leq_Z$ that satisfies $\{C_1, .. C_n\}$ can exist. This is a cyclic impossibility theorem.

Now suppose we attempt to escape $T_C$ by looking for a partially ordered notion of "better" $\leq_{Z'}$ with a comparison function $Z' : \mathbb{W \times W}  \rightarrow \{<, >, =, \stackrel{?}{=}\}$. By necessity, the uncertain case is symmetric: $a \stackrel{?}{=} b \rightarrow b \stackrel{?}{=} a$.

In some instances some of the requirements $x \overset{C_i}{\leq_{Z'}} y$ of some constraint $C_i$ will be failed, but only weakly: they will result in incomparability $x \stackrel{?}{=}_{Z'} y$ rather than violation, $x >_{Z'} y$. We call this **uncertain satisfaction** of a constraint.

**Theorem:** A cyclic impossibility theorem $T_C$ can be transformed into an uncertainty theorem only if two or more constraints are uncertainly satisfied.

*Proof*: We might hope that it would be possible to only uncertainly satisfy *one* of the constraints $\{C_1, C_2, .. C_n\}$, while fully satisfying all of the others. But this is impossible, and at minimum two of the constrains will be uncertain. The proof is by contradiction. Assume w.l.o.g. that $C_n$ is the only uncertainly satisfied constraint, such that

$$\label{eq:trans}
    w_1 \overset{C_1}{\leq_{Z'}} w_2 \overset{C_2}{\leq_{Z'}}... \overset{C_{n-1}}{\leq_{Z'}} w_n$$ but $$\label{eq:loop}
     w_n \overset{C_n}{\stackrel{?}{=}_{Z'}} w_1$$

However by transitivity and (), we know $w_1 \overset{C_{1..n-1}}{\leq_{Z'}} w_n$, which by the symmetry of $\stackrel{?}{=}$ contradicts (). So in order to treat $T_C$ as uncertainty theorem, at least two constraints must be uncertainly satisfied. QED.

The structure of this transformation from cyclic impossibility to uncertainty is shown in Figure  and for the two theorems introduced in Sections  and

The largest difficulty with accepting partially ordered, uncertain objectives is knowing how an agent should act when facing ethically uncertain choices. There are a few possible strategies, including: doing nothing, asking for human supervision, choosing randomly, searching harder for a clearly-preferable action, treating all of the uncertain options as equally good, or (for very capable agents) conducting research to try to break the tie in some way. None of these options are ideal. But depending on the application domain, some of them will be workable. An may be necessary for ML systems deployed in high stakes, safety-critical applications.

# Uncertainty theorems of the second kind: via learned, uncertain orders

A second way to encode uncertainty about a system's objectives, is to use what we will call an *uncertain objective function* or *uncertain ordering*. Suppose that an uncertain ordering is denoted by:

$$\mathbb{Z}_k(a, b) = P(a > b)$$

the uncertain ordering $\mathbb{Z}_k$ is a function returning the probability, based on the available prior philosophical and practical evidence $k$, that one of the states $a$ is better. Conversely,

$$1-\mathbb{Z}_k(a,b) = P(b > a) = \mathbb{Z}_k(b,a)$$

(For simplicity, we assume that if $a$ and $b$ appear exactly equally good, $\mathbb{Z}_k(a,b) = 0.5$ rather than allowing a non-zero probability of explicit equality.)

This formulation can for instance be justified by holding that there *is* some ultimately correct total ordering of states of the world, we just don't know what it is. The framing is also compatible with the claim that the correct ordering is ultimately unknowable.

Uncertain orderings are a convenient formalization if the comparison function is itself the output of a machine learning model to be trained from human guidance (ideally from a large number of people, over a large number of real or hypothetical scenarios). This formulation is also convenient for systems that do not have a clear architectural separation between abstract objectives and the predictive reasoning about the real world that is necessary to accomplish them.

Uncertain orderings are also a more flexible basis for strategies/decision rules than the partially ordered objective functions discussed in Section . If a partial order returns $\stackrel{?}{=}$, the agent is left torn and unable to act. But with with a distribution $\mathbb{Z}_k$ decisions rules such as "take an action drawn via weighted probability from those that are sufficiently likely to be best" are available.[^10] Or for systems where more supervision is available, "take the action that is most likely to be best, provided it beats the second best action by some probability margin $\delta$; otherwise, ask for supervision."

In this framing, whether a given ethical constraint or principle has been violated becomes a probabilistic matter. There will almost always be *some* probability that $\mathbb{Z}_k$ violates each constraint, unless all of the corresponding output probabilities have converged to 1 or 0 (as appropriate). Decision rules based on $\mathbb{Z}_k$ will by mathematical necessity sometimes violate the constraints, at least by inaction. But if the actions are sampled from a space of reasonably supported ones, then at least the violating actions will not systematically prioritize some objectives over others, and will be more similar to the way that wise and cautious humans react to difficult moral dilemmas.

What do we know about the values $\mathbb{Z}_k(a,b)$? Consider a series of cyclical constraints $C_1, C_2, ... C_n$ across points ${x_1,x_2, .. x_n}$, with $n \geq 3$, where $C_i$ requires $\mathbb{Z}_k(x_i,x_{{i+1} \mod n}) \approx 1$. If the uncertain ordering (or the process that generates it) has emitted values for some of the pairwise comparisons $\mathbb{Z}_k(x_i, x_{i+1})$, this imposes some bounds on the comparison that spans them, $\mathbb{Z}_k(x_1, x_n)$. For transitivity, in order for the spanning comparison to be in a given direction, at least one of the pairwise comparisons must also be in that direction. So for the simple two-step case over $x_1,x_1,x_3$:

$$\mathbb{Z}_k(x_1,x_3) \: \leq \: \mathbb{Z}_k(x_1,x_2) + \mathbb{Z}_k(x_2,x_3)$$

Note that the pairwise probabilities are additive because the bound is least tight when those probabilities are disjoint. Or more generally:

$$\mathbb{Z}_k(x_1,x_n) \: \leq \: \sum_{i = 1}^{n-1}\mathbb{Z}_k(x_i,x_{i+1})$$

This also applies in the reverse direction:

$$\begin{split}
    \mathbb{Z}_k(x_n, x_1) \: &\leq \: \sum_{i = 1}^{n-1}\mathbb{Z}_k(x_{i+1}, x_i) \\
    1 - \mathbb{Z}_k(x_1, x_n) \: &\leq \: \sum_{i = 1}^{n-1}1 - \mathbb{Z}_k(x_i, x_{i+1}) \\
    \mathbb{Z}_k(x_1, x_n) \: &\geq \: 1 - \sum_{i = 1}^{n-1}1 - \mathbb{Z}_k(x_i, x_{i+1})
\end{split}$$

So we have both upper and lower bounds for $\mathbb{Z}_k(x_1,x_n):$

$$\label{ztrans}
    1 - \sum_{i = 1}^{n-1}1 - \mathbb{Z}_k(x_i,x_{i+1}) \: \leq \: \mathbb{Z}_k(x_1,x_n) \: \leq \: \sum_{i = 1}^{n-1}\mathbb{Z}_k(x_i,x_{i+1})$$

In the special case where $x_1, x_2, ... x_n$ correspond to points ordered by the constraints $C_1,C_2... C_n$ of an $n$ step cyclic impossibility theorem, the satisfaction of $C_i$ means $\mathbb{Z}_k(x_i,x_{i+1 \mod n}) \approx 1$ and the probability of violation $\mathbb{Z}_k(x_{i +1 \mod n}, x_i) \approx 0$.

We might ask, what is the lowest probability that we can obtain for any of these chances of violating a constraint? Or in other words, what is the lower bound $B$ that can be achieved on the odds of each such possible violation?

$$\begin{split}
    B = \min_{\mathbb{Z}_k} &\max_{i=1}^{n-1} \{1 - \mathbb{Z}_k(x_n,x_1), 1-\mathbb{Z}_k(x_1, x_2), \:..\:, \: 1 - \mathbb{Z}_k(x_i,x_{i+1}), ..\}\\
      = \min_{\mathbb{Z}_k} &\max_{i=1}^{n-1} \{\mathbb{Z}_k(x_1,x_n), 1-\mathbb{Z}_k(x_1, x_2), \:..\:, \: 1 - \mathbb{Z}_k(x_i,x_{i+1}), ..\}\\
\end{split}$$

Then by applying the constraint from (), we get: $$B \geq \min_{\mathbb{Z}_k} \max_{i=1}^{n-1} \{1 - \sum_{j = 1}^{n-1}1 - \mathbb{Z}_k(x_j,x_{j+1}), 1-\mathbb{Z}_k(x_1, x_2), \:..\:, \: 1 - \mathbb{Z}_k(x_i,x_{i+1}), ..\}$$

By symmetry, we know that this bound will be minimized when all the values $\mathbb{Z}_k(x_i,x_{i+1 \mod n})$ are equal to a single value $z$:

$$\begin{split}
    B \geq \min_z &\max\{1-(n-1)(1- z), 1-z, \:..\:,\:1-z, ..\}\\
    B \geq \min_z &\max\{2 -n + (n - 1)z  , 1-z\}
\end{split}$$

Since in the domain $z \in [0,1]$ the function $f(z) = 2 - n + (n - 1)z$ monotonically increasing from $2 - n$, and the function $g(z) = 1-z$ is monotonically decreasing from 1, and the functions intersect, the minimax is satisfied at that intersection:

$$\begin{split}
1 -z &= 1 -z + 1 - n + nz \\
0 &= 1 - n + nz \\
z &= \frac{n - 1}
      {n}
\end{split}$$

At which point we have:

$$\begin{split}
    B \geq& 1 - \frac{n-1}{n} \\
    B \geq& \frac{1}{n}
\end{split}$$

This bound is a minimum uncertainty theorem for uncertain objective functions: when confronted with a $n$-step cyclic ethical impossibility theorem, no choice of uncertain ordering can reduce the probabilities of constraint violation so that they are all below $\frac{1}{n}$.

In practice, systems that estimate which of two outcomes are better than another could emit values like $\mathbb{Z}_k$ that violate these constraints, but in such circumstances the outputs could no longer be interpreted as probabilities over a well-defined ordering of states, and the agent would exhibit self-contradictory, non-transitive preferences.

# Further Work

We shown the existence of ethical uncertainty theorems for objectives formulated either as partial orders over states, or probability distributions over total orderings. Other formalizations are possible and deserve investigation.

Where constraints are sourced from human intuition, there is a question of how they should be interpreted, prioritized and kept in data structures. For instance, rather than treating all constraints as equally important, and all violations of constraints as equally serious, it might be better to learn or specify weightings for each constraint, to measure the *degree* to which a given action violates a constraint, and to reconstruct ethical uncertainty theorems in such a framework.

Alternatively, when building AI systems that learn their objectives from the combination of observed world states and human feedback, it may be computationally inconvenient to require the agent to produce a probability distribution over total orderings at each moment, or even to draw samples from such an object. Instead, learned objectives may be viewed as a set of pairwise comparisons that are too sparse and unstable to be taken as a commitment to any global total order (or probability distribution over global total orders). In such a frame, the goal of transitive preferences and avoidance of cycles is an aspiration, and may require both record keeping and "regret" for past actions that now seem sub-optimal due to subsequent learning of objectives. The appropriate framing of ethical uncertainty theorems in such settings would be productive further work.

# Lessons and Conjectures for Creating Aligned AI

## Uncertainty, pluralism, and instrumental convergence

Many of the concerns in the literature about the difficulty of aligning hypothetical future AGI systems to human values are motivated by the risk of "instrumental convergence" of those systems — the adoption of sub-goals that are dis-empowering of humans and other agents . The crux of the instrumental convergence problem is that given almost any very specific objective,[^11] the chance that other agents (eg, humans, corporations, governments, or other AI systems) will use their agency to work against the first agent's objective is high, and it may therefore be rational to take steps or adopt a sub-goal to remove those actors' agency.

We believe that the emergence of instrumental subgoals is deeply connected to moral certainty. Agents that are not completely sure of the right thing to do (which we believe is an accurate summary of the state of knowledge about ethics, both because of normative impossibility and uncertainty theorems, and the practical difficulty of predicting the consequences of actions) are much more likely to tolerate the agency of others, than agents that are completely sure that they know the best way for events to unfold. This appears to be true not only of AI systems, but of human ideologies and politics, where totalitarianism has often been built on a substructure of purported moral certainty .

This leads us to propose a conjecture about the relationship between moral certainty and instrumentally convergent subgoals:

**Totalitarian convergence conjecture:** *powerful agents with mathematically certain, monotonically increasing, open-ended objective functions will adopt sub-goals to disable or dis-empower other agents in all or almost all cases*.[^12]

A second conjecture is the converse of the first:

**Pluralistic non-convergence conjecture:** *powerful agents with mathematically uncertain objectives will not adopt sub-goals to disable or dis-empower other agents unless those agents constitute a probable threat to a wide range of objectives.*

## Conclusion

We have shown that impossibility theorems in ethics have implications for the design of powerful algorithmic systems, such as high-stakes AI applications or sufficiently rigid and rule-based bureaucracies. We showed that such paradoxes can be avoided by using uncertain objectives, such as partial orders or probability distributions over total orders; we proved uncertainty theorems that place a minimum bound on the amount of uncertainty required. Some previously proposed ethical theories (such Broome's vague neutral-level theory ) appear to satsify these bounds.

In the light of these results, we believe that machine learning researchers should avoid using totally ordered objective functions or loss functions as optimization goals in high-stakes applications. Systems designed that way appear to be suffering from an ethical "type error" in their goal selection (and action selection) code.

Instead, high-stakes systems should always exhibit uncertainty about the best action in some cases. Further study is warranted about the advantages of various probabilistic decision rules to handle such uncertainty, and about whether other mathematical models of uncertainy are better alternatives to the two models examined in this paper. Further research could also be productive on the relationship between ethical certainty and various observed and predicted pathological behaviour of AI systems. We proposed two conjectures on this topic, the Totalitarian Convergence Conjecture and the Pluralistic Non-Convergence Conjecture.

# Acknowledgments

The author would like to thank the late Aaron Swartz for having originally inspired this work and first attempted to make progress on this question; Toby Ord for many valuable conversations and a detailed framing of the problem; Rohin Shah for suggesting a simple proof strategy for the uncertainly satisfied bound; and Amanda Askell for providing valuable research assistance, and presenting an early poster version of this work at NIPS 2017, Daniel Ford for spotting issues and making valuable suggestions on the uncertain ordering bound. Thanks also to Dylan Hadfield-Menell, Laurent Orseau, Anca Dragan, and Stuart Armstrong for helpful suggestions and corrections on earlier versions of this work. This work was partially supported by a grant from the Open Philanthropy Project to EFF.

[^1]: The views expressed in this paper are those of the author alone, and should not be taken as positions of the Partnership on AI, its members, or the Electronic Frontier Foundation.

[^2]: There are related and more fruitful questions like, "how should self-driving cars make trade-offs between time saved by driving more aggressively and accident risk?" See for instance .

[^3]: The creation of autonomous weapons systems may well be an extremely bad idea (see, eg , but one of the main arguments made by their proponents is that such systems will be able to make ethical decisions in real world equivalents of trolley situations at least as well as if not better than human pilots, and some military-funded work in that direction is already underway (see, for instance, ).

[^4]: The "maximise average wellbeing" objective fails this dramatically; in that case the proposed positive addition can be just below the welfare level of the original group, but in general for other objective functions the proposed addition may be only slightly above 0.

[^5]: This is most notably true in the economics literature , though there are now some efforts from the machine learning direction too .

[^6]: Although one philosophical concern is that methods like CIRL may already commit us to more 'person-affecting' views within ethics by pooling the preferences of existing agents. This could lead to the implementation of principles that satisfy undesirable axioms like dictatorship of the present .

[^7]: For instance, if a linear weighting is chosen that appears to make sense for the present population of the world, a change in technology that allowed a much larger population might quickly cause the average wellbeing to cease affecting the preferred outcome in any meaningful way.

[^8]: See `https://pdollar.github.io/toolbox/classify/softMin.html`. `softmin` is the counterpart to the more widely discussed `softmax` function. It prioritises whichever of its inputs is currently the smallest, is not totally unresponsive to increases in the other inputs.

[^9]: Arrhenius considered this direction of interpretation , but with much more uncertainty than turned out to be necessary:

    > \[A person arguing that some actions in the Mere Addition Paradox are neither right nor wrong\] could perhaps motivate her position by saying that moral theory has nothing to say about cases that involve cyclical evaluations and that lacks a maximal alternative since these are beyond the scope of moral theory. Compare with the theory of quantum mechanics in physics and the impossibility of deriving the next position and velocity of an electron from the measurement of its current position and velocity (often expressed by saying that it is impossible to determine both the position and velocity of an electron).

    It turns out that only two of the edges in the cycle need to be uncertain. Note that Broome's vague neutral-level theory is one plausible example of an uncertain social welfare function.

[^10]: This has been termed a "satisficing" or "quantilized" decision rule . A similar rule is the *contextual-*$\epsilon$*-greedy* reinforcement learning policy proposed in ; both of these decision rules combine explore/exploit tradeoffs with the notion that certain actions might be too costly or dangerous to take as experiments in certain situations. In *contextual-*$\epsilon$*-greedy* this is formalized via annotating certain states as "high level critical situations", whereas threshholding probabilities views all `(state, action)` pairs as potentially critical.

[^11]: Any open-ended and non-trivial objective appears to be vulnerable to instrumental convergence. Bostrom argues that objectives that are bounded rather than open-ended also lead to instrumental convergence if there is any probability that they will not be achieved (eg, the goal "make exactly one million paperclips" could cause an agent to be paranoid that it hasn't counted exactly correctly) though this failure mode is probably easy to avoid by rounding the estimated probability of success to some number of digits, or imposing a small cost in the objective function for additional actions or resource consumption.

[^12]: The qualifier "almost all" requires some further specification. It requires that the objective not be finely tuned in some very intricate way that builds in preservation of all other agents. It is unclear if such fine-tuning is either properly definable or possible.


abstract: Semantic parsing is the task of mapping natural language to logic form. In question answering, semantic parsing can be used to map the question to logic form and execute the logic form to get the answer. One key problem for semantic parsing is the hard label work. We study this problem in another way: we do not use the logic form any more. Instead we only use the schema and answer info. We think that the logic form step can be injected into the deep model. The reason why we think removing the logic form step is possible is that human can do the task without explicit logic form. We use BERT-based model and do the experiment in the WikiSQL dataset, which is a large natural language to SQL dataset. Our experimental evaluations that show that our model can achieves the baseline results in WikiSQL dataset.
author: Tong Guo Huilin Gao
title: Table2answer: Read the database and answer without SQL

# Introduction

One way to construct a question-answering system over database is leveraging the semantic parsing. Semantic parsing is a task that transform the natural language to logic form which computer can execute. Transforming from natural language to SQL (NL2SQL) is kind of semantic parsing task. The generated SQL can be executed in the database system to can the answer from the database. In recent years, deep learning techniques is applied to semantic parsing. But deep learning need large amount of labeled data, when the parameter number is very large. The logic form of natural language is also very hard to label, compared to other natural language processing (NLP) tasks such as text classification or sentence similarity. There are some works solve the hard labeling problem in weak supervision methods. Our work try to solve this problem in another way. The upper level view of the problem of semantic parsing for question answering is retrieve the answer from the database when given a question. Human can solve this problem even without an explicit logic form. Human can read the schema or columns' info in the database and answer the question. We think the deep model can integrate the logic or reasoning modules like or other deep model to search on the database without an explicit logic form. We present our idea in Fig. 1. and Fig. 2.

We use BERT-based to implement our idea. BERT is a new language representation model, which stands for Bidirectional Encoder Representations from Transformers. Pre-trained word representations on a large (unlabeled) language corpus, such as , have shown promising results in a lot of NLP tasks. BERT is also a pre-trained deep model which use large amount of plain text to pre-train. As a result, the pre-trained BERT representations can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and text classification.

Our main contributions in this work are two-fold. First, we introduce our idea of finding the answer from database with out semantic parsing to solve the problem of hard label work for semantic parsing. Second, we use BERT-based model to implement our idea and achieve a baseline level experiment result. The code is available. [^1]

# Task Description

In the NL2SQL without SQL task (Table2answer), given a question and a database table, the deep model needs to find the answer in the database table. The question is described as a sequence of word tokens: $Q = \{w_1,w_2,...,w_n\}$, where $n$ is the number of words in the question, and the table is described as a sequence of column names or headers $H=\{h_1,h_2,...,h_m\}$, where $m$ is the number of columns in the table. Each table contains a number of rows which contains the answer or cells to the question. The answer for the model is the pointers to the table cells. We denote the cells as $T = \{c_1,c_2,...,c_{r \times m}\}$, where r is the number of rows. Note that each $c_{r \times m}$ is not one word in the table, but each $c_{m \times r}$ is one cell in the table. In experiment, we concatenate the word embeddings in a cell to represent one cell. The cell representation is the input of the transformer layers of BERT.

We now describe the WikiSQL dataset , a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables from Wikipedia. We present an example in Fig. 3.

We extract the question, answer index and table content from the WikiSQL dataset and construct the dataset for Table2answer. One table corresponding to several questions with answers in the table. For an elementary consideration, we only extract the data case which SQL of question only contains one condition in the WHERE clause. We leave it as future work which solves all kinds of questions with SQL in the WikiSQL. We present an example in Fig. 4. Also, Our model works under the condition that the table which contains the answer is determined. In other words, our model need not predict the exact table in the all tables. We use other methods to find the exact table in industry application.

# Model

In this section, we describe the details of our BERT-based model to solve the problem of question answering over database without semantic parsing. We present the overall solution for the Table2answer problem in Fig. 5. The reason and inspiration to use BERT-based model is that we want to leverage the attention info between question and table header to point the exact cell in the table.

We follow the BERT convention of data input format for encoding the natural language question together with the headers and cells of the table. We use \[SEP\] to separate between the cells and the question. We average word embedding of one cell for the representation of one cell. At last, each headers $H=\{h_1,h_2,...,h_m\}$, table cells $T = \{c_1,c_2,...,c_{r \times m}\}$ and question $Q = \{w_1,w_2,...,w_n\}$ is encoded as following:

$[CLS], h_1, h_2, ..., h_m, c_1, c_2, ..., c_{r \times m}, [SEP], w_1, w_2, ..., w_n, [SEP]$

In the SQuAD machine comprehension task, we input the paragraph and question to the model and find the answer string in the paragraph. And our Table2answer task, we input the question, headers and table cells to find the answer cell in the table. The two tasks are very similar in this perspective. So we append one pointer after the output of the BERT main module. It is different from machine comprehension task because we only have one cell to point as we have simplified the dataset. We leave it as future work to choose more than one cell as answer.

# Experiments

In this section, we present more details of the model and the evaluation on the dataset. Pre-trained BERT models (BERT-Base-Uncased) are loaded and fine-tuned with Adam optimizer with learning rate $5*10^5$. The batch size is 16. We use the origin BERT tokenizer with the same vocabulary of BERT-Base-Uncased. We fix the parameters of 1-9 layers of BERT-Base and fine-tune the last 3 layers, as we observe that fine-tuning all the layers do not get a better evaluation result. Our neural network model is implemented in TensorFlow.

## Data augmentation

We randomly shuffle the rows of all the tables and get a training dataset of 503881 data and test dataset of 1874 data. Note that the answer cell index is corresponding to the shuffled rows. We have not shuffled the columns of table as we observe bad result of it. See Tab. 1. for detail.

| Training Data Size | Test Accuracy |
|:-------------------|:--------------|
| 76301              | 20.3$\%$      |
| 321536             | 47.2$\%$      |
| 503881             | 54.0$\%$      |

The evaluation of our experiment for data augmentation.

| Model                                | Test Accuracy |
|:-------------------------------------|:--------------|
| transformer baseline                 | 11.0$\%$      |
| Our model                            | 54.0$\%$      |
| Our model without data augmentation  | 17.7$\%$      |
| Our model without position embedding | 20.5$\%$      |

The evaluation of our experiment. Our baseline is a transformer without pre-training. As there may be same answers in different cells, we consider the final word match as the accuracy.

## Evaluation

We evaluate our model on the dataset that extract from WikiSQL. The results are presented in Tab. 2. The training accuracy is around 96$\%$ and we leave it as future work to further improve the result on the test dataset. We also do the experiment just the same as SQuAD machine comprehension task. That is, we concatenate all the words in the table cells and append two pointer after BERT for the start index and end index. And the result for this kind of data feeding methods is 2$\%$-3$\%$ lower.

# Conclusion

In this paper, in order to solve the problem that the labeling work for semantic parsing is too hard, we introduce our idea that inject the reasoning part into the deep model to remove the logic form step for question answering. We think that human can do the logic operation even without the SQL so we believe our idea will work. Then We design the BERT-based model and achieve the baseline results in the sub-WikiSQL dataset. It is trained end-to-end and can retrieve the answer directly. The dataset for table2answer is simpler than WikiSQL and there will be a lot of work to research.

[^1]: <https://github.com/guotong1988/table2answer>


abstract: Business process automation is a booming multi-billion-dollar industry that promises to remove menial tasks from workers' plates – through the introduction of autonomous agents – and free up their time and brain power for more creative and engaging tasks. However, an essential component to the successful deployment of such autonomous agents is the ability of business users to monitor their performance and customize their execution. A simple and user-friendly interface with a low learning curve is necessary to increase the adoption of such agents in banking, insurance, retail and other domains. As a result, proactive chatbots will play a crucial role in the business automation space. Not only can they respond to users' queries and perform actions on their behalf but also initiate communication with the users to inform them of the system's behavior. This will provide business users a natural language interface to interact with, monitor and control autonomous agents. In this work, we present a multi-agent orchestration framework to develop such proactive chatbots by discussing the types of skills that can be composed into agents and how to orchestrate these agents. Two use cases on a travel preapproval business process and a loan application business process are adopted to qualitatively analyze the proposed framework based on four criteria: performance, coding overhead, scalability, and agent overlap.
author: Yara Rizk, Abhishek Bhandwalder, Scott Boag, Tathagata Chakraborti,
**Vatche Isahagian, Yasaman Khazaeni, Falk Pollock, Merve Unuvar**
IBM Research AI
75 Binney Street
Cambridge, Massachusetts 02142
email@example.com, firstname.lastname@example.com, email@example.com, email@example.com,
email@example.com, email@example.com, firstname.lastname@example.com, email@example.com
bibliography: BPA_chatbot.bib
title: A Unified Conversational Assistant Framework
  for Business Process Automation

# Introduction

Automation made its way into manufacturing plants in the early 1900s, however comparatively, its integration into the service industry has been slow. The service industry first took advantage of automation through customer care chatbots which routed incoming customer calls to the appropriate human operators. Gradually, these robotic answering machines improved and began to replace services provided by some human operators by walking customers through simple steps to resolve problems and escalating to humans only when necessary. As enabling technologies improved, so have these customer care chatbots which can now carry out more elaborate conversations with the customers.

Beyond chatbots, little automation was incorporated in the services industry despite the plethora of automation opportunities. The emergence of Robotic Process Automation (RPA) led to the next breakthrough. RPA developed robotic software agents capable of automating simple tasks within a business process such as form filling and data mining. RPAs reduced the cost of deploying autonomous agents but covered a small portion of automatable tasks, namely simple repetitive tasks. This led to the emergence of Business Process Automation (BPA) which supersedes RPA's scope. In addition to task automation, BPA looked to automate decision making, data management, content digitization, and workflow improvement. BPA relies on various artificial intelligence technologies including optical character recognition (OCR) systems and decision making models to automate business process tasks. As state-of-the-art algorithms become more accurate and reliable - deep learning OCR has achieved super-human accuracy levels - automation in service industry will become more cost-effective.

Nevertheless, wide-spread deployment of autonomous systems has still lagged behind expectations. Trustworthiness, coding overhead, scalability, algorithm life cycle, technical expertise and user experience have been a few challenges that have stood in the way of more successful integration into business process solutions. In this work, we claim that a proactive multi-agent conversational framework can be that one stop shop that resolves many of these challenges. However, multiple research questions at the intersection of automation and business processes must be address first. A conversational interface reduces the necessary technical expertise to interact, monitor and customize the framework, but many algorithms do not have a conversational interface. How can we effectively integrate such algorithms into a conversational framework with minimal coding overhead? How can such independently developed (and potentially incompatible) functions coexist and cooperate within the same ecosystem? How could the user access all of the automation options in the framework with minimal interface and protocol hoping? The goal is not to create a natural language interaction for every part of the process but a single interactive interface. If it is accessing different documents, running OCR on a new application, checking our database or setting up alerts on specific events in our system, one would prefer to interact with a single interface which is conveniently conversational in this scenario.

We believe an orchestrator and agent composition approach with a specific pipeline would provide solutions to these questions. We present a multi-agent orchestration framework that consists of heterogeneous agents such as dialog, informational retrieval, task execution, and alerting agents and an orchestration layer that consists of three components (scorer, selector and sequencer) that would control the execution of the agents in the framework. As a proof of concept, we create two instances of this assistant for a travel preapproval business process and a loan application business process – deployed in a sandbox environment – and discuss how this framework can address these challenges.

# Related Work

## Business Process Automation

Task automation has driven the digital transformation in enterprises with the emergence of RPA as a light-weight approach for automating repetitive tasks. RPA is a paradigm that seeks to automate the "mouse-click" in user interfaces to relieve human resources of repetitive tasks. RPA adopts an outside-in approach that requires minimal changes to legacy software ; they have been applied to various sectors including accounting , auditing , human resources , banking , public administration and energy sectors .

Multiple approaches have been adopted in the development of RPAs. proposed an RPA solution for document flow automation in a debt collector business process. They used deep learning OCR and classification to control the path of the document through the business process. User behavior was captured to train a self-learning agent capable of identifying relationships between tasks and deploying RPAs based on the learned rules . This approach, known as form-to-rule, relied on first-order logic to deduce the rules. Another branch of literature focused on identifying RPA-eligible tasks automatically. Leopold et al. relied on natural language processing of business process descriptions . They adopted supervised machine learning to classify instances into one of three classes.

As the scope of RPA increases to include more complex tasks, RPA agents incorporate other technological advancements such as OCR for document digitization, and data mining for content collection and management to successfully automate complex tasks. These advancements have achieved the best performance when adopting deep learning models . However, these models are computationally expensive to train and have seen slow adoption despite their super-human accuracy due to their lack of transparency and explainability.

Beyond RPA, BPA also seeks for automated decision making within business processes. The field of automated planning has been considered a prime candidate to enable more advanced levels of automation in business processes . Planning algorithms can be used to automatically generate business process models, adapt the business process' behavior based on unforeseen circumstances, and autonomously perform conformance checking.

End-to-end BPA has yet to find success in real-world deployments despite the ubiquity of computing devices and the advancements in software technology. Trust and reliability have been two of the main challenges preventing wide-spread adoption of automation approaches. With business stakeholders lacking the technical skills to monitor autonomous agents and intervene when they do not operate as expected, they have been reluctant to adopt this technology. Providing a natural language interface to allow such users to interact with autonomous agents may be the key to increase the adoption of automation in service enterprises.

## Conversational Agents

Natural language is the main communication modality in business enterprises. From conversations to written documents, forms and emails, understanding and generating natural language can significantly facilitate the integration of various autonomous agents into domains where users do not speak the language of computers. Conversational agents have been developed for a wide range of applications using a plethora of machine learning techniques that have resulted in agents with increasing degrees of sophistication. In this section, we will focus on chatbots developed for enterprise applications and specifically those used in conjunction with RPA. Two approaches to chatbot development have been mainly adopted in the literature: 1) building domain specific chatbots by relying on natural language understanding and generative machine learning models, and 2) training deep learning models on data samples . While the former requires more overhead to develop, it has been the more widely adopted approach for enterprise chatbots due to its predictable and controllable behavior.

In enterprises, chatbots were first adopted for customer support which involved answering customers' questions about the services, products, location etc. of businesses . Beyond question answering bots, a shopping bot was developed to increase the sales of shops by answering questions from customers and even providing custom prices for products . A delivery bot was also developed for food delivery services that reduced the effort required from customers to order pizza . While only automating 30% of the processes with these bots, the bot developers have recently begun investigating the incorporation of RPA to increase the percentage of automation. Matthies et al. developed a chatbot for agile software development teams which analyzed commits in version control systems to provide insights into the teams' performance. Kalia et al. proposed an approach to generate a conversational agent based on dialog trees given a business process flow . Even though they provide a systematic way to develop chatbot that were sound, significant overhead and domain knowledge were necessary.

Furthermore, these approaches have focused on designing a single conversational agent capable of answering frequently asked questions and performing tasks in a process such as determining the price of products in the shopping bot. Expanding the scope of their operation would require significant coding overhead by developers who understand the existing system. Also, reusing existing agents may not be possible due to compatibility reasons. In this work, we propose a conversation agent framework based on the multi-agent system paradigm as a potential solution to these issues. Instead of designing a single powerful agent, multiple specialized agents with narrower scopes can be independently designed and then integrated into a multi-agent system to act as a single conversational agent from the user's perspective.

# Methodology

## Overall Framework

The framework for the proactive conversational assistant for BPA consists of three main components: skills, agents, and an orchestrator, as shown in Fig. . Skills perform well defined tasks; they require a set of inputs to produce a set of outputs. Agents are composed of skills that fall into three main categories: understand skills, act skills, and respond skills. An execution pattern for the skills must be provided; it can be as simple as a static sequential script or as complex as a full execution graph. The orchestrator coordinates between agents and it determines which agent or agents must execute to successfully respond to a user. When a user enters an utterance, this natural language sentence is forwarded to all the agents in the assistant by the orchestrator after potentially preprocessing the input. The orchestrator requests a preview of each agent's response and their confidence score. This score depicts how confident an agent is in its ability to respond to the user. Depending on the adopted orchestration pattern, the orchestrator selects one or more agents to execute and return their responses to the user.

## Skills

Skills are atomic functions, the building blocks of agents. They are categorized according to their role within an agent and their effect on the state of the world. Skills can have one of three roles: understand, act and respond. Furthermore, skills can either be world changing or non-world changing.

### Understand skills

Serving as the entry point to an agent, understand skills identify the user's intent. In the conversational setting, understand skills are generally intent and entity recognition skills but can also be as simple as a key-phrase detection. They may take a user's natural language utterance and classify it into one of many predefined intents that the agent can handle or check if a specific phrase designated to trigger the agent is there. They may also annotate any entities they recognize within the input. In a non-dialog setting, understand skills are event triggered skills that watch for specific events and state changes in the system. These events could be a user logging in or a change in database.

### Act skills

Act skills process the user input to produce an output that can be used to generate a response. We distinguish two main types of act skills: world changing and non-world changing. World changing skills perform actions that modify the state of the world or have side-effects outside the confines of the orchestrator world. Examples of such skills include credit score checking skills, email sending skills, etc. Non-world changing skills do not have any side effects. Examples include email reading skills, weather checking skills, etc. Act skills in the business process world can perform various tasks or activities. One task is gathering information by querying databases or making API calls to information sources. Decision points are a set of activities that move the business process forward. Another task that moves the business process forward is submitting applications or forwarding gathered information to the next activity.

### Respond skills

Respond skills create the response that will be returned to the user by the conversational assistant. The skills take the output of act skills and process them to form natural language responses to the user; these could be full flesh natural language generation to template responses given different outputs of the other skills. Other modalities of responses can also be considered such as visualization of the output in the form of plots or images.

## Agents

Individually, the skills may not be able to perform complex tasks required by the activities within a business process. Instead of increasing the scope of individual skills by implementing more complex functionality within them, composing them into agents capable of completing these activities is more effective. This modular approach simplifies debugging of skills and agents and increases the reusability of skills by making them as domain agnostic as possible.

We define a contract for the agents in order to facilitate their orchestration within a single assistant. The agents expect to receive an input message or utterance and the current context which represents the state of the system. The agent is capable of producing a preview response that does not cause any world-changing actions to be invoked, an execute response that results from fully executing its pipeline and may cause the world to change, an updated context once the agent has executed, and a confidence in its response.

Multiple approaches can be adopted to compose skills into agents. Creating the execution pipeline can be done statically where a software developer writes a script that explicitly defines the order in which skills must be executed and directs the flow of data by routing inputs and outputs. In goal-oriented non-deterministic planning based composition , software developers declaratively define the inputs and outputs of skills and the end goal of the agent. Then, a planning engine composes an execution tree using the subset of available skills that will lead to the goal. While this approach allows independent developers to include their skills in a catalog, composing skills into agents in an effective way requires a minimum level of compatibility between the skills, including the adoption of a unified vocabulary for input/output variables.

Agents are composed of an understand skill, a set of act skills, and a respond skill. Depending on the functionality of the agent, none, one or multiple act skills may be included in the composition. Next, we discuss four main types of agents based on their functionality. While many more agents can be created, we consider the ones that would be most relevant and commonly used in BPA.

### Dialog Agents

Users may just need to inquire about certain topics and do not require any action to be taken by the assistant. In such scenarios, a dialog agent, composed of an understand skill for intent and entity recognition and a respond skill to answer the user's query. Examples of such agents include chit-chat agents, FAQ or "help" agents, etc. They can be implemented in dialog tree type services. These agents do not change the world; their preview mode and execute mode are the same.

### Information Retrieval Agents

Unlike dialog agents, information retrieval agents must connect to an external service (such as making an API call or querying a database) in order to produce their answer to a user's query as opposed to having the answering within the agent implementation. As a result, such agent compositions consist of three types of skills: understand skill that determine the intent of the user and annotate any entities that may be useful, one or more act skills that queries the appropriate service to produce its answer, and a respond skill that appropriately constructs the response returned to the user. In general, most agents within this category do not contain skills that change the world. However, their preview and execute modes may vary for computational/latency reasons. In some situations, querying does change the world; for example, retrieving the credit score of a loan applicant is considered as world changing as it will have a negative impact on the applicant if queried a large number of times. In other cases, simply incurring a cost, for example monetary or in the form or using up API calls that are limited, to query a database can be modeled as world changing to minimize the cost incurred by the system.

### Task Execution Agents

Beyond querying for information, the business user may want the assistant to perform tasks within the business process. Task execution agents are capable of moving the business process forward by submitting application, making decisions at decision points and such. These agents will change the world and therefore may require a preview and execute mode based on the type of orchestrator adopted. These agents are composed of an understand skill, a set of act skills and a respond skill.

### Alerting Agents

Setting up alerts to receive notifications triggered by the occurrence of specific events allows business users to monitor the business process in an asynchronous way. Like the other agents, these agents are also composed of an understand skill, a set of act skills and a respond skill. However, their understand skill does not necessarily have to be a natural language understanding skill. However, setting up custom alerts would require an intent and entity recognition understand skill. These agents consist of a watcher skill on the system that may trigger the alert, a notification generation skill and a communication channel skill that sends the notification over the appropriate communication channel (e.g. email, SMS, slack...). A natural language understanding skill can be added to allow the user to customize the trigger and settings through dialog. If that is the case, a Natural Language to Query (NLQ) skill would be necessary to convert natural language trigger definitions to structured, well-defined queries on the database that may trigger the alert and is watched by the daemon. If an NLQ skill is not available, hard-coded conversions must be present. However, this approach restricts the number of alerts that can be created. NLQ provides a broader range of triggers that can be set. It also allows the creation of domain agnostic alert agents which would be more portable and require minimal coding overhead to deploy in new domains, a desirable feature in the BPA field. In this paper, we distinguish between alerts and notifications. The former is the occurrence of an event that causes the conditions of the trigger to be satisfied. The latter is the actual message sent to the user when an alert happens.

## Orchestration

Combining these diverse agents into a single assistant requires a domain agnostic orchestration layer capable of determining which subset of agents are best suited to respond to the user's utterance. Various orchestration frameworks with diverse characteristics can be adopted to achieve this goal. Stateless orchestrators do not keep track of the state in a centralized location. Instead the context variables (state) are passed between the orchestrator and agents. Stateful orchestrators on the other hand, maintain the state of the system in a centralized location. The computational cost of data transfer in the former approach increases with the number of context variables maintained in the state. As a result, system designers tend to minimize the number of variables in their state resulting a decrease in the amount of information agents have access to. Stateful orchestrators, however, require a central device to maintain the state which could be costly from a deployment and scalability perspective.

Apriori orchestrators decide which agents should execute by considering the user input and what they know about agent capabilities. Posterior orchestrators pass the inputs to the agents requesting their responses, then decide which agents to select. In this case, preview and execute modes of agents must be available for those agents that perform world changing actions. Orchestrators also expect agents to provide a confidence in their responses which are used as a feature in their decision making algorithm.

Unlike dialog managers, orchestrators do not require knowledge of the dialog to select agents. Dialog managers are assumed to be part of those agents that need it. For example, dialog agents would have managers that keep track of which node of a tree the conversation is in or perform dialog disambiguation and co-reference resolution.

The 3S orchestrator is an example of a posterior orchestrator that relies on the agent's confidence, among other metrics, to determine which agents should respond. Its pipeline consists of three main steps: scoring, selecting, and sequencing agents .

### Scorer

When the orchestrator receives the user's utterance, it forwards this input to all agents and requests a preview of their response. Once the agents return their preview responses and their confidence score, the scorer performs additional computations to obtain the final scores per agent. An identity function scorer simply forwards the scores as is to the selector. A scorer may normalize scores, especially since they could have been produced by very diverse agents that compute their scores differently. A more sophisticated scorer could enhance the agents' confidences with other measures such as conversation stickiness, dialog depth (within a dialog tree) and agent engagement. A Bayesian model that adapts to user feedback could also be adopted.

### Selector

Once the scorer computes the final scores per agent, it forwards them to the selector that must select one or more agents that will respond to the user. It can be as simple as a *Top 1 (or Top K) score* selector that picks the agent (or K agents) with the highest score. More complex selection criteria could be adopted such as heuristic rules, learned selection models from labeled user data, online reinforcement learning models trained on user feedback, or a combination of multiple approaches. Once the selector determines the subset of agents that must execute, the orchestrator requests their execution responses. Various machine learning algorithms can be adopted to improve the robustness of the selector as the noise in the score increases. In a supervised learning domain, labels can represent agents that are the output of classifiers and the input to the classifier would be a feature vector consisting of the agents' scores and/or user utterances. In a reinforcement learning domain, actions can represent agents and the environment produces a reward when the correct agent is selected. For both approaches, deep learning models can be trained if enough data is provided to further improve the selection model.

### Sequencer

After receiving the agents' execution responses, a sequencer determines the order in which these responses will be presented to the user. If a *Top 1 score* selector was adopted in the previous stage, then no sequencing is necessary. The response is directly outputted to the user through the conversational interface. However, when a subset of agents has responded, a sequencer could adopt different approaches to order these responses: e.g. descending score, heuristic rules (e.g. greeting agent must always respond before other agents), supervised or reinforcement learning models trained on labeled data and user feedback.

# Use Case – 1: Travel Preapproval Application

We will now detail the implementation of a conversational assistant (in Slack) for two personas (employees, and managers/directors) in a simplified travel preapproval application process using the framework described so far.

| **Conversation** | **Responding agent** |
|:---|:---|
| Manager: Hello | Chit-Chat |
| *Travelbot*: Hi there |  |
| Manager: Retrieve the number of accepted papers authored by John Smith | Publication Query |
| *Travelbot*: The number of accepted papers by John Smith is 7 |  |
| Manager: Approve John Smith's request | Business Process |
| *Travelbot*: John Smith's application has been approved | Task Execution |

Conversation samples

## Travel Preapproval Business Process

We adopt a simplified business process revolving around travel preapprovals generally used across many corporations. We consider the scenario of researchers/employees requesting travel funds to attend research conferences. The travel preapproval process, shown in Fig. , was implemented in a business process management software.

Employees submit a travel preapproval request to attend a conference by filling in specific information into a form, shown in Fig. (a). Once submitted, the application is forwarded to the employee's manager who can approve/reject the application (Fig. (b)) or request additional information/changes to the application (Fig. (c)). If approved by the manager, the application is forwarded to the director who makes the final decision on the travel request.

## Agents in *Travelbot*

*Travelbot* contains multiple agents that enable it to respond to a broad range of user requests. The 3S orchestrator coordinates between the composed agents to respond to the user or execute any actions requested by the user. Table includes a sample interaction between *Travelbot* and a manager.

### Chit-Chat Agent

A simple dialog agent, Chit-Chat is implemented as a question answering dialog tree that is capable of answering utterances related to greetings (e.g. "hello", "how are you", "goodbye", etc.). It can also introduce the assistant answering questions such as "who are you" and "how can you help me". Additional intents and dialog trees are also included to make the assistant more human-like and improve the user experience – e.g. "what are you doing", "tell me a joke", etc. This agent does not have any act skills; it does not cause any side effects when executed and hence has identical preview and execute modes.

### Publication Query Agent

This information retrieval agent queries a database containing the publications of employees and conference information. In the adopted use case, this agent can be used to auto-fill the information in the application form related to travel justification and conference location and dates. It can also be used by managers and directors to obtain information necessary for making decisions. For example, travel requests would be approved for employees presenting their work at the conference as first authors. This agent makes this information accessible to managers through a conversational interface and eliminates the need for them to switch to a different interface. The static composition of this agent consists of a Natural Language Understanding (NLU) understand skill, an act skill capable of retrieving employee information from an employee directory that would be used by the publication query skill (second act skill) to retrieve publication/conference information. Finally, the respond skill generates the final response that will be returned to the user.

### Business Process Data Query Agent

Another informational retrieval agent, the business process data query agent allows managers and directors to query the existing travel request applications. Managers and directors can ask questions like "How many pending travel requests are in my queue?" or "How many applications have been submitted by employee X?". Such queries factor into the decision making process and allow decision makers to make more informed decisions. This agent is composed of three skills: an NLU understand skill, a data query act skill, and a respond skill.

### Business Process Task Execution Agent

This agent allows users to move the business process forward by requesting the assistant to submit an application on their behalf (e.g. "Submit an application to AAAI 2020 for me.") or approve/reject/send back an application (e.g. "Reject Jack's application to ICML 2019."). The agent composition is more complex in this case and consists of four act skills that may or may not be invoked depending on the recognized intent. Like the other agents, this agent also has an intent and entity recognition understand skill and a respond skill. Its act skills are employee directory query, paper directory skill, travel pricing skill and business process task execute act skill. The pricing skill is used auto-fill the "requested amount" field in Fig. (a).

### Travel Estimation Agent

This agent can respond to inquiries about travel such as – "What is the cheapest flight from BOS to SFO leaving on 2019/12/01 and returning on 2019/12/07?". This information retrieval agent consists of an intent and entity recognition understand skill, a travel pricing skill which uses the Amadeus API to estimate the cost of flights and hotels, and a respond skill that formulates the answer returned to the user. Managers or directors can use this agent to verify the "requested amount" field submitted by the employee.

### Visualization Agent

Beyond natural language responses, the visualization agent provides the assistant with the ability to return visual responses. This is useful in situations where information retrieval agents return data that is easier to digest in a plot form. Therefore, the user can ask the assistant to plot the results. This agent is composed of an intent and entity recognition understand skill to if and which plot the user wants and what variables to use if multiple options exist. A plotter skill converts the plottable data to an image of a bar graph or doughnut chart or other visualization format that is then forwarded to the user using the respond skill. Due to the stateless implementation of the orchestrator, the data is persisted for a finite number of iterations in the context that is passed between agents the orchestrator. If the visualization agent is invoked by a user utterance, it will look for plottable data in the context. If this data does not exist, then a default response of "There is no data to be plotted." is returned to the user.

### Alerting using Natural Language Query

*TravelBot* allows the user to create and customize alerts based on user's data-store. The system accepts user input and if it is determined that intent is the creation of alert, the sentence is then passed through an NLQ module , , . The NLQ module can express any natural language sentence into a data query which can run on top of many different types of data-stores (ElasticSearch, SQL, etc). The query from NLQ module is then stored and a daemon runs in the background monitoring these queries. If there is any change in the data store (Addition of new row, changes in values of an existing row) and the query output changes, an alert is generated and sent to the user on the communication channel. The system determines the communication channel for a specific user through conversation. Alerts could be configured to send notifications via Slack, email or SMS using the Twilio API.

# Use Case – 2: Loan Application

Using the same multi-agent framework, we also develop a loan officer chatbot for a loan application business process. We include: 1) a data query agent capable of answering questions related to existing applications, 2) a business rules agent that can provide insights into whether an application will be accepted or rejected, 3) a visualization agent capable of representing data in different formats, 4) an alerts agent that sends notifications when changes to loan applications occur, and 5) a content analyzer that can process loan documents and extract information from them. Sample conversations with this chatbot are included in Table and Fig. .

| **Conversation** | **Responding agent** |
|:---|:---|
| Loan Officer: What is the total loan amount for borrowers with credit score more than 500? | Data query |
| *Loanbot*: The sum value is 137368000.0 |  |
| Loan Officer: Who are the top 3 borrowers with average amount more than 10000 | Data query |
| *Loanbot*: These are the value: 1). average of J. Smith is 584917\$, 2). average of V. Doe is 575692\$, 3). average of Y. Doe is 557615\$ |  |
| Loan Officer: List all borrowers with yearly income more than 50000 but credit score less than 150 | Data query |
| *Loanbot*: Total records found are 82. Here is the link: $<$url$>$ |  |
| Loan Officer: Plot the bar chart per yearly income | Visualization |
| *Loanbot*: $<$image$>$ (Fig. ) |  |
| Loan Officer: Find the top 5 borrowers in terms of total amount of loans | Data query |
| *Loanbot*: The result for your query is: $<$csv$>$ (Fig. ) |  |
| Loan Officer: Could you process an application requesting a loan of 3000\$? | Business rules |
| *Loanbot*: What is the credit score? |  |
| Loan Officer: 400 | Business rules |
| *Loanbot*: What is the annual salary (in USD) |  |
| Loan Officer: 5000 | Business rules |
| *Loanbot*: In how many months will the loan be paid back? |  |
| Loan Officer: 12 | Business rules |
| *Loanbot*: High risk loan. This loan request should not be approved |  |

Sample conversation with LoanBot

# Challenges

Multiple challenges have hindered the deployment of autonomous conversational agents in BPA domains. We focus on a few of these challenges and discuss how the proposed framework can overcome them.

### Coding Overhead

The enterprises that need chatbots often do not have the technical expertise or resources to maintain and improve the framework. Thus, it is important that the system requires minimal coding overhead to deploy in new domains, expand the scope of the system and improve its performance by adding new agents during its lifetime. Our framework assumes that independent developers, whether domain experts or software developers, have created the autonomous agents but require that they abide by the adopted input-output contracts. This reduces the coding overhead to integrate the agents. Furthermore, the 3S orchestrator is robust enough to handle the noise in the agent confidence which is a byproduct of independent agent developers.

### Scalability

Expanding the scope and capabilities of a conversational agent requires significant developer effort and may result in performance degradation. In the proposed framework, integration is trivial but naively adding agents may result in performance issues if the orchestrator incorrectly selects agents. Adopting an orchestration model that learns from previous interactions and changes in the system would improve the overall performance of the assistant.

### Agent Overlap

As the number of agents in the framework increases, some agent functionality and knowledge may overlap. This may cause conflicts among agents when attempting to respond to users and degrade overall performance. As a result, it is important to model or quantify agent overlap and incorporate it within the orchestrator model to insure that the orchestrator will select the best agent to complete the task of responding to the user. Adopting a learning selector in the orchestrator pipeline can lead to better selection models that can handle agent overlap and prevent performance degradation.

### Multi-user Support

Giving users the ability to compose their personalized assistants requires the framework to support multiple users. While this can be resolved by creating a session per user, it provides an opportunity to increase the available data from which the orchestrator can learn better execution models. Therefore, the proposed framework should consider approaches to learn from multiple users' data while maintaining privacy and security of the data.

### Access Control

With different personas in business processes, some agents may not be accessible to specific users. For example, with the *Travelbot*, employees should not have access to the "approve/reject" functionality. Our framework requires that these restrictions are implemented inside the agent. When the user submits a request, the orchestrator forwards this request to all agents and expects them to decline execution if the user does not have the required access.

# Conclusion

In this paper, we presented a multi-agent framework to develop a conversational assistant supporting multiple capabilities such as querying data through natural language, autonomously executing tasks in a business process, alerting users of changes in the business process and visualizing data in various forms based on the user's requests. An orchestrator capable of combining independently developed skills with minimal overhead is the key to effectively deploy more powerful conversational assistants. While many challenges remain such as scalability and agent overlap, preliminary results motivate follow on research.