abstract: We describe an architecture for spoken dialogue interfaces to semi-autonomous systems that transforms speech signals through successive representations of linguistic, dialogue, and domain knowledge. Each step produces an output, and a meta-output describing the transformation, with an executable program in a simple scripting language as the final result. The output/meta-output distinction permits perspicuous treatment of diverse tasks such as resolving pronouns, correcting user misconceptions, and optimizing scripts.
author: Manny Rayner, Beth Ann Hockey, Frankie James
Research Institute for Advanced Computer Science
Mail Stop 19-39, NASA Ames Research Center
Moffett Field, CA 94035-1000
{manny,bahockey,firstname.lastname@example.com
bibliography: rialist.bib
title: A Compact Architecture for Dialogue Management Based on Scripts and Meta-Outputs

# Introduction[^1]

The basic task we consider in this paper is that of using spoken language to give commands to a semi-autonomous robot or other similar system. As evidence of the importance of this task in the NLP community note that the early, influential system SHRDLU was intended to address just this type of problem. More recent work on spoken language interfaces to semi-autonomous robots include SRI's Flakey robot and NCARAI's InterBOT project . A number of other systems have addressed part of the task. CommandTalk , Circuit Fix-It Shop and TRAINS-96 are spoken language systems but they interface to simulation or help facilities rather than semi-autonomous agents. Jack's MOOse Lodge takes text rather than speech as natural language input and the avatars being controlled are not semi-autonomous. Other researchers have considered particular aspects of the problem such as accounting for various aspects of actions . In most of this and other related work the treatment is some variant of the following. If there is a speech interface, the input speech signal is converted into text. Text either from the recognizer or directly input by the user is then converted into some kind of logical formula, which abstractly represents the user's intended command; this formula is then fed into a command interpreter, which executes the command.

We do not think the standard treatment outlined above is in essence incorrect, but we do believe that, as it stands, it is in need of some modification. This paper will in particular make three points. First, we suggest that the output representation should not be regarded as a logical expression, but rather as a program in some kind of scripting language. Second, we argue that it is not merely the case that the process of converting the input signal to the final representation can sometimes go wrong; rather, this is the normal course of events, and the interpretation process should be organized with that assumption in mind. Third, we claim, perhaps surprisingly, that the first and second points are related. These claims are elaborated in Section .

The remainder of the paper describes an architecture which addresses the issues outlined above, and which has been used to implement a prototype speech interface to a simulated semi-autonomous robot intended for deployment on the International Space Station. Sections  and  present an overview of the implemented interface, focussing on representational issues relevant to dialogue management. Illustrative examples of interactions with the system are provided in Section . Section  concludes.

# Theoretical Ideas

## Scripts vs Logical Forms

Let's first look in a little more detail at the question of what the output representation of a spoken language interface to a semi-autonomous robot/agent should be. In practice, there seem to be two main choices: atheoretical representations, or some kind of logic.

Logic is indeed an excellent way to think about representing static relationships like database queries, but it is much less clear that it is a good way to represent *commands*. In real life, when people wish to give a command to a computer, they usually do so via its operating system; a complex command is an expression in a scripting language like CSHELL, Perl, or VBScript. These languages are related to logical formalisms, but cannot be mapped onto them in a simple way. Here are some of the obvious differences:

-   A scripting language is essentially imperative, rather than relational.

-   The notion of temporal sequence is fundamental to the language. "Do $P$ and then $Q$" is not the same as "Make the goals $P$ and $Q$ true"; it is explicitly stated that $P$ is to be done first. Similarly, "For each $X$ in the list $(A\ B\ C)$, do $P(X)$" is not the same as "For all $X$, make $P(X)$ true"; once again, the scripting language defines an order, but not the logical language[^2].

-   Scripting languages assume that commands do not always succeed. For example, UNIX-based scripting languages like CSHELL provide each script with the three predefined streams `stdin`, `stdout` and `stderr`. Input is read from `stdin` and written to `stdout`; error messages, warnings and other comments are sent to `stderr`.

We do not think that these properties of scripting language are accidental. They have evolved as the result of strong selectional pressure from real users with real-world tasks that need to be carried out, and represent a competitive way to meet said users' needs. We consequently think it is worth taking seriously the idea that a target representation produced by a spoken language interface should share many of these properties.

## Fallible Interpretation: Outputs and Meta-outputs

We now move on to the question of modelling the interpretation process, that is to say the process that converts the input (speech) signal to the output (executable) representation. As already indicated, we think it is important to realize that interpretation is a process which, like any other process, may succeed more or less well in achieving its intended goals. Users may express themselves unclearly or incompletely, or the system may more or less seriously fail to understand exactly what they mean. A good interpretation architecture will keep these considerations in mind.

Taking our lead from the description of scripting languages sketched above, we adapt the notion of the "error stream" to the interpretation process. In the course of interpreting an utterance, the system translates it into successively "deeper" levels of representation. Each translation step has not only an input (the representation consumed) and an output (the representation produced), but also something we will refer to as a "meta-output": this provides information about how the translation was performed.

At a high level of abstraction, our architecture will be as follows. Interpretation proceeds as a series of non-deterministic translation steps, each producing a set of possible outputs and associated meta-outputs. The final translation step produces an executable script. The interface attempts to simulate execution of each possible script produced, in order to determine what would happen if that script were selected; simulated execution can itself produce further meta-outputs. Finally, the system uses the meta-output information to decide what to do with the various possible interpretations it has produced. Possible actions include selection and execution of an output script, paraphrasing meta-output information back to the user, or some combination of the two.

In the following section, we present a more detailed description showing how the output/meta-output distinction works in a practical system.

# A Prototype Implementation

The ideas sketched out above have been realized as a prototype spoken language dialogue interface to a simulated version of the Personal Satellite Assistant (PSA; ). This section gives an overview of the implementation; in the following section, we focus on the specific aspects of dialogue management which are facilitated by the output/meta-output architecture.

## Levels of Representation

The real PSA is a miniature robot currently being developed at NASA Ames Research Center, which is intended for deployment on the Space Shuttle and/or International Space Station. It will be capable of free navigation in an indoor micro-gravity environment, and will provide mobile sensory capacity as a backup to a network of fixed sensors. The PSA will primarily be controlled by voice commands through a hand-held or head-mounted microphone, with speech and language processing being handled by an offboard processor. Since the speech processing units are not in fact physically connected to the PSA we envisage that they could also be used to control or monitor other environmental functions. In particular, our simulation allows voice access to the current and past values of the fixed sensor readings.

The initial PSA speech interface demo consists of a simple simulation of the Shuttle. State parameters include the PSA's current position, some environmental variables such as local temperature, pressure and carbon dioxide levels, and the status of the Shuttle's doors (open/closed). A visual display gives direct feedback on some of these parameters.

The speech and language processing architecture is based on that of the SRI CommandTalk system . The system comprises a suite of about 20 agents, connected together using the SRI Open Agent Architecture (OAA; ). Speech recognition is performed using a version of the Nuance recognizer . Initial language processing is carried out using the SRI Gemini system , using a domain-independent unification grammar and a domain-specific lexicon. The language processing grammar is compiled into a recognition grammar using the methods of ; the net result is that only grammatically well-formed utterances can be recognized. Output from the initial language-processing step is represented in a version of Quasi Logical Form , and passed in that form to the dialogue manager. We refer to these as *linguistic level* representations.

The aspects of the system which are of primary interest here concern the dialogue manager (DM) and related modules. Once a linguistic level representation has been produced, the following processing steps occur:

-   The linguistic level representation is converted into a *discourse level representation*. This primarily involves regularizing differences in surface form: so, for example, "measure the pressure" and "what is the pressure?" have different representations at the linguistic level, but the same representation at the discourse level.

-   If necessary, the system attempts to resolve instances of ellipsis and anaphoric reference. For example, if the previous command was "measure temperature at flight deck", then the new command "lower deck" will be resolved to an expression meaning "measure temperature at lower deck". Similarly, if the previous command was "move to the crew hatch", then the command "open it" will be resolved to "open the crew hatch". We call the output of this step a *resolved discourse level representation*.

-   The resolved discourse level representation is converted into an executable script in a language essentially equivalent to a subset of CSHELL. This involves two sub-steps. First, quantified variables are given *scope*: for example, "go to the flight deck and lower deck and measure pressure" becomes something approximately equivalent to the script

        foreach x (flight_deck lower_deck)
          go_to $x
          measure pressure
        end

    The point to note here is that the `foreach` has scope over both the `go_to` and the `measure` actions; an alternate (incorrect) scoping would be

        foreach x (flight_deck lower_deck)
          go_to $x
        end
        measure pressure

    The second sub-step is to attempt to optimize the plan. In the current example, this can be done by reordering the list `(flight_deck lower_deck)`. For instance, if the PSA is already at the lower deck, reversing the list will mean that the robot only makes one trip, instead of two.

-   The final step in the interpretation process is *plan evaluation*: the system tries to work out what will happen if it actually executes the plan. (The relationship between plan evaluation and plan execution is described in more detail in Section ). Among other things, this gives the dialogue manager the possibility of comparing different interpretations of the original command, and picking the one which is most efficient.

## How Meta-outputs Participate in the Translation

The above sketch shows how context-dependent interpretation is arranged as a series of non-deterministic translation steps; in each case, we have described the input and the output for the step in question. We now go back to the concerns of Section . First, note that each translation step is in general fallible. We give several examples:

-   One of the most obvious cases arises when the user simply issues an invalid command, such as requesting the PSA to open a door $D$ which is already open. Here, one of the meta-outputs issued by the plan evaluation step will be the term

        presupposition_failure(already_open(D));

    the DM can decide to paraphrase this back to the user as a surface string of the form "$D$ is already open". Note that plan evaluation does not involve actually executing the final script, which can be important. For instance, if the command is "go to the crew hatch and open it" and the crew hatch is already open, the interface has the option of informing the user that there is a problem without first carrying out the "go to" action.

-   The resolution step can give rise to similar kinds of meta-output. For example, a command may include a referring expression that has no denotation, or an ambiguous denotation; for example, the user might say "both decks", presumably being unaware that there are in fact three of them. This time, the meta-output produced is

        presupposition_failure(
            incorrect_size_of_set(2,3))

    representing the user's incorrect belief about the number of decks. The DM then has the possibility of informing the user of this misconception by realizing the meta-output term as the surface string "in fact there are three of them".

    Ambiguous denotation occurs when a description is under-specified. For instance, the user might say "the deck" in a situation where there is no clearly salient deck, either in the discourse situation or in the simulated world: here, the meta-output will be

        presupposition_failure(
            underspecified_definite(deck))

    which can be realized as the clarification question "which deck do you mean?"

-   A slightly more complex case involves plan costs. During plan evaluation, the system simulates execution of the output script while keeping track of execution cost. (Currently, the cost is just an estimate of the time required to execute the script). Execution costs are treated as meta-outputs of the form

        cost(C)

    and passed back through the interpreter so that the plan optimization step can make use of them.

-   Finally, we consider what happens when the system receives incorrect input from the speech recognizer. Although the recognizer's language model is constrained so that it can only produce grammatical utterances, it can still misrecognize one grammatical string as another one. Many of these cases fall into one of a small number of syntactic patterns, which function as fairly reliable indicators of bad recognition. A typical example is conjunction involving a pronoun: if the system hears "it and flight deck", this is most likely a misrecognition of something like "go to flight deck".

    During the processing phase which translates linguistic level representations into discourse level representations, the system attempts to match each misrecognition pattern against the input linguistic form, and if successful produces a meta-output of the form

        presupposition_failure(
            dubious_lf(<Type>))

    These meta-outputs are passed down to the DM, which in the absence of sufficiently compelling contrary evidence will normally issue a response of the form "I'm sorry, I think I misheard you".

# A Compact Architecture for Dialogue Management Based on Scripts and Meta-Outputs

None of the individual functionalities outlined above are particularly novel in themselves. What we find new and interesting is the fact that they can all be expressed in a uniform way in terms of the script output/meta-output architecture. This section presents three examples illustrating how the architecture can be used to simplify the overall organization of the system.

## Integration of plan evaluation, plan execution and dialogue management

Recall that the DM simulates evaluation of the plan before running it, in order to obtain relevant meta-information. At plan execution time, plan actions result in changes to the world; at plan evaluation time, they result in *simulated* changes to the world and/or produce meta-outputs.

Conceptualizing plans as scripts rather than logical formulas permits an elegant treatment of the execution/evaluation dichotomy. There is one script interpreter, which functions both as a script executive and a script evaluator, and one set of rules which defines the procedural semantics of script actions. Rules are parameterized by execution type which is either "execute" or "evaluate". In "evaluate" mode, primitive actions modify a state vector which is threaded through the interpreter; in "execute" mode, they result in commands being sent to (real or simulated) effector agents. Conversely, "meta-information" actions, such as presupposition failures, result in output being sent to the meta-output stream in "evaluate" mode, and in a null action in "execute" mode. The upshot is that a simple semantics can be assigned to rules like the following one, which defines the action of attempting to open a door which may already be open:

    procedure(
     open_door(D),
     if_then_else(status(D, open_closed, open),
       presupposition_failure(already_open(D)),
       change_status(D, open_closed, open)))

## Using meta-outputs to choose between interpretations

As described in the preceding section, the resolution step is in general non-deterministic and gives rise to meta-outputs which describe the type of resolution carried out. For example, consider a command involving a definite description, like "open the door". Depending on the preceding context, resolution will produce a number of possible interpretations; "the door" may be resolved to one or more contextually available doors, or the expression may be left unresolved. In each case, the type of resolution used appears as a meta-output, and is available to the dialogue manager when it decides which interpretation is most felicitous. By default, the DM's strategy is to attempt to supply antecedents for referring expressions, preferring the most recently occurring sortally appropriate candidate. In some cases, however, it is desirable to allow the default strategy to be overridden: for instance, it may result in a script which produces a presupposition failure during plan evaluation. Treating resolution choices and plan evaluation problems as similar types of objects makes it easy to implement this kind of idea.

## Using meta-outputs to choose between dialogue management moves

Perhaps the key advantage of our architecture is that collecting together several types of information as a bag of meta-outputs simplifies the top-level structure of the dialogue manager. In our application, the critical choice of dialogue move comes after the dialogue manager has selected the most plausible interpretation. It now has to make two choices. First, it must decide whether or not to paraphrase any of the meta-outputs back to the user; for example, if resolution was unable to fill some argument position or find an antecedent for a pronoun, it may be appropriate to paraphrase the corresponding meta-output as a question, e.g. "where do you mean?", or "what do you mean by 'it'?". Having all the meta-outputs available together means that the DM is able to plan a coherent response: so if there are several meta-outputs which could potentially be worth paraphrasing, it typically realizes only the most important one.

Second, if interpretation was able to produce a well-formed plan, the DM currently has the three options of executing it, paraphrasing it back to the user as a confirmation question, or doing nothing. Once again, this decision often requires global information about what has happened during the interpretation process. For example, knowing that plan evaluation showed that the plan would take significant time to execute makes it more plausible that the user would prefer to receive a confirmation.

# Examples

This section presents a commented sample dialogue with the implemented PSA interface demo chosen to illustrate the points discussed in the previous sections. We focus in particular on the role that meta-outputs play in interpretation. Text preceded by USER represents spoken utterances from the user. Text preceded by PSA represents synthesized spoken responses from the PSA. Text inside square brackets describes the actions carried out by the PSA.

1.  The PSA can move to various locations and measure environmental factors. In the first exchange, reference resolution is used to interpret the phrase "all three decks" as referring to the lower deck, mid deck and flight deck. Plan optimization finds the most efficient way to order the locations, and paraphrases the resulting plan back to the user.

    USER:

    :   Go to all three decks and measure carbon dioxide.

    PSA:

    :   I will move to flight deck, mid deck and then lower deck and I will measure carbon dioxide level, okay?

    USER:

    :   Okay. *\[PSA moves to flight deck\]*

    PSA:

    :   The carbon dioxide level at the flight deck is one percent. *\[PSA moves to mid deck\]*

    PSA:

    :   The carbon dioxide level at the mid deck is one percent. *\[PSA moves to lower deck\]*

    PSA:

    :   The carbon dioxide level at the lower deck is one percent.

2.  The next example involves resolution of ellipsis. The relevant context is recovered from the preceding dialogue.

    USER:

    :   Do the same for the pilot's seat.

    PSA:

    :   I will move to the pilot's seat and I will measure carbon dioxide level, okay?

    USER:

    :   Right. *\[PSA moves to pilot's seat\]*

    PSA:

    :   The carbon dioxide level at the pilot's seat is one percent.

3.  Resolution of ellipsis can involve filling in unspecified semantic parameters to a syntactically complete command. It is not meaningful to talk about a sensor reading without reference to the place at which that reading was carried out; in the following example, resolution assumes that the location in question is the one most recently mentioned in the dialogue.

    Note also that since the PSA does not have to move in order to carry out the command, and hence can execute it immediately, the DM reasonably decides to skip the confirmation question. This decision is made based on the meta-output encoding the estimated cost of executing the script.

    USER:

    :   What was the carbon dioxide level at fifteen oh five according to the fixed sensors?

    PSA:

    :   According to the fixed sensors, at fifteen oh five the carbon dioxide level at the pilot's seat was one percent.

4.  The system encounters a user misconception, which results in a "presupposition failure" meta-output being produced during reference resolution. It consequently decides to alert the user to this, rather than attempt to carry out the command.

    USER:

    :   Close both doors.

    PSA:

    :   There are in fact three of them.

5.  The user utterance includes a pronoun, "it". The system resolves "it" to "the crew hatch", and issues an appropriate paraphrase in order to confirm that this is what the user meant.

    USER:

    :   Go to crew hatch and close it.

    PSA:

    :   I will go to crew hatch and I will close crew hatch, okay?

    USER:

    :   Yeah. *\[PSA moves to crew hatch and closes it\]*

6.  The user asks the system to close the door, but it is unclear which door they are referring to. An "ambiguous reference" meta-output is produced by reference resolution, resulting in the DM issuing a clarification question.

    USER:

    :   Close the door.

    PSA:

    :   Which door do you mean?

7.  The user response is successfully interpreted by reference resolution as an instance of ellipsis. Trying to close the crew hatch a second time however causes a presupposition failure meta-output to be generated during plan evaluation. The system informs the user of this and takes no action.

    USER:

    :   The crew hatch.

    PSA:

    :   The crew hatch is already closed.

8.  The user presents three locations in a suboptimal order. Plan evaluation finds a more efficient ordering.

    USER:

    :   Move to storage lockers, commander's seat and flight deck and measure temperature.

    PSA:

    :   I will move to flight deck, commander's seat and then storage lockers and I will measure temperature, okay?

    USER:

    :   Sure.
        *\[PSA moves to flight deck, commander's seat and storage lockers in that order and reports temperature at each location\]*

9.  The user asks the system to repeat the previous command. Since the PSA is now starting from a different position, its most efficient plan is no longer the same, and it informs the user of this.

    USER:

    :   Do that again.

    PSA:

    :   I will move to storage lockers, commander's seat and then flight deck and I will measure temperature, okay?

    USER:

    :   Yes. *\[PSA moves to storage lockers\]*

    PSA:

    :   The temperature at the storage lockers is 19.9 degrees Celsius. *\[PSA starts moving to commander's seat\]*

10. The user interrupts the PSA in the middle of the plan, and asks it to go back to its last stopping place. Since "stop" may involve a potentially hazardous condition, the interface obeys immediately, without attempting to confirm first.

    USER:

    :   Stop. *\[PSA stops\]*

    USER:

    :   Go back. *\[PSA returns to storage lockers\]*

# Summary

We have described a generic architecture for spoken language dialogue interfaces to semi-autonomous agents, based on the standard notion of translating to successively deeper levels of representation. The novel aspects of our scheme center around two ideas: first, that the final output representations are best conceptualized not as logical expressions but rather as programs in a scripting language; second, that steps in the translation process should produce not only a simple output, but also meta-information describing how the output was produced. We have presented examples suggesting how several apparently diverse types of dialogue behavior can be captured simply within our framework, and outlined a prototype implementation of the scheme.

[^1]: This paper also appears in the proceedings of the Sixth International Conference on Applied Natural Language Processing, Seattle, WA, April 2000.

[^2]: In cases like these, the theorem prover or logic programming interpreter used to evaluate the logical formula typically assigns a conventional order to the conjuncts; note however that this is part of the *procedural* semantics of the theorem prover/interpreter, and does not follow from the declarative semantics of the logical formalism.


author: V. Ossenkopf$\!$, M. Röllig, R. Simon, N. Schneider, Y. Okada, J. Stutzki, M. Gerin, M. Akyilmaz, D. Beintema, A.O. Benz, O. Berne, F. Boulanger, B. Bumble, O. Coeur-Joly, C. Dedes, M.C. Diez-Gonzalez, K. France, A. Fuente, J.D. Gallego, J.R. Goicoechea , R. Güsten, A. Harris, R. Higgins, B. Jackson, C. Jarchow, C. Joblin, T. Klein, C. Kramer, S. Lord, P. Martin, J. Martin-Pintado, B. Mookerjea, D.A. Neufeld, T. Phillips, J.R. Rizzo, F.F.S. van der Tak, D. Teyssier, H. Yorke
title: HIFI observations of warm gas in DR21: Shock versus radiative heating[^1]

The molecular gas in the DR21 massive star formation region is known to be affected by the strong UV field from the central star cluster and by a fast outflow creating a bright shock. The relative contribution of both heating mechanisms is the matter of a long debate. By better sampling the excitation ladder of various tracers we provide a quantitative distinction between the different heating mechanisms. HIFI observations of mid-$J$ transitions of CO and HCO$^+$ isotopes allow us to bridge the gap in excitation energies between observations from the ground, characterizing the cooler gas, and existing ISO LWS spectra, constraining the properties of the hot gas. Comparing the detailed line profiles allows to identify the physical structure of the different components. In spite of the known shock-excitation of H$_2$ and the clearly visible strong outflow, we find that the emission of all lines up to $\ga2$ THz can be explained by purely radiative heating of the material. However, the new *Herschel/HIFI* observations reveal two types of excitation conditions. We find hot and dense clumps close to the central cluster, probably dynamically affected by the outflow, and a more widespread distribution of cooler, but nevertheless dense, molecular clumps.

# Introduction

DR21 is a deeply embedded H ii region created by the radiation from at least six OB stars . It sits within a ridge of dense molecular material that obscures the H ii-region at optical wavelengths. The embedded cluster drives a violent bipolar outflow in north-east to south-west direction. It is prominent in the 2 $\mu$m emission of vibrationally excited H$_2$, tracing hot, shocked gas and in the Spitzer 4.5 $\mu$m channel . As the cluster is located close to the eastern edge of the molecular ridge, the eastern, blue-shifted outflow expands in a blister-like fountain, while the western, red-shifted outflow is highly collimated .

Spitzer 8 $\mu$m images reveal spots of bright PAH emission with a size below 10$''$ . They represent the surfaces of high-density, UV irradiated clumps, forming photon-dominated (or photo-dissociation) regions (PDRs), transition zones from ionized and atomic gas to dense molecular gas where physics and chemistry are dominated by UV radiation from young stars . These hot and dense regions give rise to the emission of PDR tracers, such as HCO$^+$, high-$J$ CO, atomic and ionized carbon, and atomic oxygen.

An overview on the numerous existing observations in DR21 is given by and . In spite of the wealth of data, the heating of the molecular gas is still debated. The bright H$_2$ emission indicates shock heating in the outflow and the wings of the molecular lines prove its dynamical impact, but excitation models of the observed emission of CO, \[C i\], \[C ii\], and \[O i\] by and have shown that the emission of those tracers cannot be explained by shocks, but is consistent with a pure UV heating, i.e., PDR physics.

To quantify the heating of the gas, we analyze spectra taken with the *HIFI* instrument on board the *Herschel Space Observatory* during the performance verification campaign. Observations of high-$J$ HCO$^+$ transitions trace hot material, ionized by UV radiation or X-rays . Hot water lines are produced in shocked gas , and observations of CO isotopes around $J=10$ close the gap in the excitation ladder between ground-based observations and existing ISO data, allowing to obtain a full picture of the temperature distribution.

In Sect. 2 we present the observational data. Sect. 3 compares the measured line profiles to distinguish different components based on their velocity distribution. In Sect. 4 we provide a model for the emission, supporting the PDR character of the source, and discuss the results in Sect. 5.

# Observations

## HIFI observations

All spectra presented here were obtained in performance verification observations for the HIFI instrument. As their main goal was to demonstrate the functionality and performance of the different observing modes, the spectra were taken with a large variety of observing modes and strategies. Consequently, every spectrum was taken in a slightly different manner. All observational parameters are summarized in Table .

Summary of the used HIFI observational data

transition
$\nu_{\rm line}$

HPBW

observing mode1

$t_{\rm int,source}$

rms

[GHz]
″

[s]
[K]

HCO+ 6-5
535.062
40
OTF map
16
0.04

HCO+ 12-11
1069.694
21
FSW spectral scan
150
0.1

13CO 10-9
1101.350
21
LC spectral scan
270
0.08

C18O 10-9
1097.163
21
LC spectral scan
270
0.08

1900.537
12
DBS raster map
14
1

20
convolved+binned
112
0.2

40
convolved+binned
420
0.1

$^1$ DBS = dual-beam-switch, OTF = On-The-Fly, FSW = frequency-switch, LC = load-chop, OFF position = 20h37m10s, 42$^\circ$`<!-- -->`{=html}37$'$`<!-- -->`{=html}00$''$

Most observations where single-point observations towards the central position of the DR21 H ii region at RA=20h39m01.1s, DEC=42$^\circ$`<!-- -->`{=html}19$'$`<!-- -->`{=html}43.0$''$ (J2000). Fully sampled maps were only obtained in the \[CII\] line. Data were taken with the wideband spectrometer (WBS) at a resolution of 1.1 MHz, corresponding to 0.2 km/s (at 1900 GHz) – 0.7 km/s (at 500 GHz). The \[CII\] data were rebinned to a velocity resolution of 0.45 km/s to improve the signal to noise.

## Complementary data

ISO Long Wavelength Spectrometer $43-197$ $\mu$m grating scans were obtained for the DR21 central position from the ISO Data Archive (TDT 15200786). Integrated line intensities were extracted for \[O i\] at 63 and 145 $\mu$m and the CO 14–13 to 17–16 transitions. Mid-$J$ CO lines of the DR21 region were mapped with the KOSMA 3 m submm telescope . We use the lines of CO and $^{13}$CO from $J=3$-2 to 7-6, which have been observed at native angular resolutions from 80$''$ to 40$''$. The HCO$^+$ 1-0, H$^{13}$CO$^+$ 1-0, and HCO$^+$ 3-2 observations were taken with the IRAM 30 m telescope . Native angular resolutions at the 1-0 and 3-2 transition frequencies are 28$''$ and 9$''$, respectively.

## Beam size effects

For a direct comparison of the different data sets we smoothed the available data to the coarsest common angular resolution of 80$''$, matching that of the CO 3-2 KOSMA beam and the lowest frequency ISO observations. This is impossible for the single-point HIFI observations. Moreover, the *Herschel* beam varies between 40$''$ HPBW for the HCO$^+$ line at 535 GHz and 20$''$ for $^{13}$CO at 1100 GHz. Those data were corrected for the different beam filling by estimating the source size from the HCO$^+$ 3-2 line as a PDR tracer with quite compact emission (20$''$ to 30$''$ in diameter). Successively convolving from the native angular resolution of 9$''$ to the angular resolution of the *Herschel* data, and further to the final, smoothed spectra at 80$''$, we obtained scaling factors for the HIFI spectra, being 0.5 when going from 40$''$ to 80$''$ and 0.33 when going from 20$''$ to 80$''$. The maps obtained in \[CII\] allow a direct smoothing to a resolution of 40$''$ (see Table ). Beyond that size, the same scaling factor as above was applied.

# Line profiles

The molecular ridge including the DR21 H ii region has an intrinsic LSR velocity of -3 km s$^{-1}$. The quiescent material is visible in narrow absorption lines of NH$_3$ and H$_2$CO . A second velocity component at 8-10 km s$^{-1}$ is known to be associated with the W75N complex. It appears in emission in CO and \[C i\] , as a narrow absorption feature in HCO$^+$ 1-0 , and has a very broad velocity distribution in the H i 21 cm absorption . The wings of the low-$J$ CO lines trace outflow velocities down to about $-$`<!-- -->`{=html}20 km s$^{-1}$ for the eastern, blister outflow and up to $\approx 10$ km s$^{-1}$ for the western outflow.

Figure shows the profiles of the HIFI spectra of CO isotopes, HCO$^+$ and \[C ii\]. All lines peak at about -4 km s$^{-1}$. The CO and HCO$^+$ lines have a similar shape, but the \[C ii\] line shows an additional broad blue wing extending down to -30 km s$^{-1}$. This indicates that the warm molecular material is slightly blue-shifted relative to the cold gas and that the \[C ii\] emission is not only originating from that warm gas, but also from the ionized wind in the blister outflow.

Figure  compares the shapes of the CO and HCO$^+$ isotope lines with complementary ground-based measurements towards the same positions. We show only a few selected transitions as, e.g., the data for the CO 7-6 or 4-3 lines provide no additional information. All CO isotopic lines up to 7-6 are roughly symmetric, centered at the ridge velocity of $-3$ km s$^{-1}$. The lines of the main isotope are heavily self-absorbed with the absorption dip marking line center. The 10-9 lines, tracing hotter material, are slightly asymmetric and shifted to -4 km s$^{-1}$. All HCO$^+$ lines, except H$^{13}$CO$^+$ 1-0, have a profile very similar to the 10-9 lines of the CO isotopes indicating that HCO$^+$ is mainly abundant in the heated layer and less in the overall molecular material. The H$^{13}$CO$^+$ 1-0 line has a deviating profile centered at -2 km s$^{-1}$ tracing global infall . CO 3-2 and HCO$^+$ 1-0 show an additional feature at the W75N complex velocity.

To better understand the exact velocity distribution in the source, we plot some additional HIFI lines towards the same position in Fig. . The 1097 GHz line of hot water matches the rest of the PDR tracers. The foreground at -3 km s$^{-1}$ is visible in absorption in the ground state transitions of water, but in emission in a nearby OFF position for \[C ii\]. At velocities of about +2 km s$^{-1}$, we find the warm gas facing the redshifted western outflow, apparent as a secondary peak in the ground state water and CH$^+$ lines and as a shoulder in $^{13}$CO and the hot water line. Finally, we can clearly identify the cold W75N component at $\approx -9$ km s$^{-1}$.

For the warm gas we can distinguish two velocity components – a blue shifted one related to the blister outflow at $-4$ km s$^{-1}$ and a second one, at 2 km s$^{-1}$, related to the western outflow. This is consistent with the clumpy PDR geometry proposed by . In terms of line modelling, it is, however, impossible to separate the two outflow directions as their emission is overlapping to a large degree. We simply add their intensities. The \[C ii\] emission requires a special treatment because of the additional emission from the ionized outflow. To take this into account, we have obtained an integrated \[C ii\] intensity by scaling the $^{13}$CO 10-9 line profile to match the peak and the red wing of the \[C ii\] profile and ignoring the remaining blue-wing emission.

# Modelling

We use the KOSMA-$\tau$ PDR code to model the emission of PDR ensembles, representing a distribution of spherical clumps with $dN/dM \propto M^{-1.8}$ . For DR21 two ensembles with different properties had to be superimposed, a hot component, close to the inner H ii region with strong FUV illumination, but only a small fraction of the total mass, and a cooler component that fills a larger solid angle and provides the bulk of the material. Each clumpy PDR ensemble has five free parameters: the average ensemble density, $n_\mathrm{ens}$, the ensemble mass, $M_\mathrm{ens}$, the UV field strength, $\chi$ given in units of the Draine field, and the minimum and maximum mass of the clump ensemble, $[M_\mathrm{min},M_\mathrm{max}]$. In contrast to most other PDR models, we fit absolute line intensities, using the available ground-based observations, complementary ISO data, and the HIFI lines of the CO isotopes, HCO$^+$, atomic and ionized carbon, and atomic oxygen. The chemical network that has been applied in these calculations includes $^{13}$C but not $^{18}$O. The C$^{18}$O lines were scaled from the $^{13}$CO intensities with a conversion of 1:8. Simulated annealing was used to find the optimum parameter combination.

The significance of the model is limited by the fact that the clump superposition ignores mutual line shading between different clumps, i.e., optical depth effects are only considered within individual clumps. This is usually justified by the virialised velocity dispersion between different clumps, but for optically very thick and broad lines some correction is needed. To estimate the effect we have computed the optical depth for the bulk of the individual clumps. This is of the order of unity or below for the majority of the observed transitions, reaches values up to ten for $^{13}$CO and HCO$^+$ transitions up to $J=4$, but exceeds ten for the CO main isotope lines up to $J=6$ and the \[O i\] 63 $\mu$m line. For the CO lines showing clear self-absorption dips, we performed a Gaussian fit to the line wings and used the integrated intensity of that Gaussian to compute the total emission including the blocked radiation from the inner clumps close to the H ii-region. As we have no spectral information for the \[O i\] line, we have no estimate for the blocked radiation in this case, so that we excluded that data point from the fit.

The best fit result is shown in Figure . The corresponding model parameters are:

Ensemble 1:

:   $\chi=1.0\,10^{5}$, $n_{\rm ens}=1.3\,10^{6}$cm$^{-3}$, $M_{\rm ens}=150~M_\odot$, $[M_{\rm min},M_{\rm max}]=[10^{-2},8\,10^1] M_\odot$

Ensemble 2:

:   $\chi=3.2\,10^{2}$, $n_{\rm ens}=1.1\,10^{6}$cm$^{-3}$, $M_{\rm ens}=830~M_\odot$, $[M_{\rm min},M_{\rm max}]=[10^{-3},10^1] M_\odot$

The two-ensemble model provides a reasonable fit to all 22 data points. The model slightly overestimates the emission for the optically thick rotational lines of $^{13}$CO and HCO$^+$ which showed no clear self-absorption dip, so that we have insufficient optical depth corrections. The reduced $\chi^2$ value of the fit amounts to 11.1, where a contribution of 9.5 results from the three optically thick low-$J$ data points of $^{13}$CO and HCO$^+$. For the remaining 19 data points we would obtain an excellent reduced $\chi^2$ value of 1.6. The topology of the $\chi^2$ function shows several side minima, but they are worse by at least a factor two. As expected, the model predicts a too high \[O i\] intensity, as it ignores that the outer clumps of the cooler ensemble block the contribution from the hot inner component.

# Discussion

The double-peak structure of the predicted CO intensities across the excitation ladder reflects the two different UV fields leading to different excitation conditions. No single-parameter ensemble can fit all data, but distinct low temperature and high temperature components are needed. The parameters of the model are in agreement with independent estimates. A UV flux of $10^5$ Draine fields corresponds to a geometrical distance of 0.06 pc, i.e. 7$''$, from the central cluster, matching the size of the PAH emission . Our clump densities match those determined by for the extended cool gas, but are slightly higher than their hot-gas density ($4\,10^5$ cm$^{-3}$). In contrast, find still somewhat higher densities for the hot gas, up to $10^7$ cm$^{-3}$. The hot ensemble mass is close to the 170 $M_\odot$ derived from early CO 7-6 observations by . The total mass of the PDR ensembles falls between the mass limits derived by from dust observations and from line radiative transfer fits.

While the existing ground-based observations provide a very good constraint on the properties of the extended cool gas, and the ISO lines show the total amount of hot gas, it is only the set of new HIFI data that puts the hot and cold distributions well apart from each other in terms of the temperature structure. While obtained cooling curves with single peaks, the new data for the 10-9 lines of the CO isotopes and the HCO$^+$ transitions force the fit to a bimodal distribution of excitation conditions. When we exclude the *Herschel* data from the model fit, we obtain a parameter set that shows a UV field that is lower by a factor ten for the hot ensemble, i.e., that would imply molecular clumps farther away from the central cluster. For the cold ensemble, the fitted UV field is also somewhat lower, while all other parameters remain similar to those from the full fit. Only with the *Herschel* data, we therefore obtain a parameter set that is consistent with the source geometry.

As the two-ensemble PDR model is able to fit all of the observed lines, we find no evidence for a shock heating of the dense gas. This is in agreement with the analysis of , explicitely excluding a shock origin of the fine-structure lines, but seems to be in contradiction with the analysis of the line profiles in Sect. that shows excited outflow material. We conclude that the material visible in the blue line wing, characterizing the blister outflow, is contained in dense clumps that are accelerated by the outflow, but that are chemically and energetically fully dominated by the UV field and not by the associated shock.

[^1]: *Herschel* is an ESA space observatory with science instruments provided by European-led Principal Investigator consortia and with important participation from NASA.


abstract: Acoustic model adaptation to unseen test recordings aims to reduce the mismatch between training and testing conditions. Most adaptation schemes for neural network models require the use of an initial one-best transcription for the test data, generated by an unadapted model, in order to estimate the adaptation transform. It has been found that adaptation methods using discriminative objective functions – such as cross-entropy loss – often require careful regularisation to avoid over-fitting to errors in the one-best transcriptions. In this paper we solve this problem by performing discriminative adaptation using lattices obtained from a first pass decoding, an approach that can be readily integrated into the lattice-free maximum mutual information (LF-MMI) framework. We investigate this approach on three transcription tasks of varying difficulty: TED talks, multi-genre broadcast (MGB) and a low-resource language (Somali). We find that our proposed approach enables many more parameters to be adapted without over-fitting being observed, and is successful even when the initial transcription has a WER in excess of 50%.
address:  Centre for Speech Technology Research, University of Edinburgh, Edinburgh EH8 9AB, UK
bibliography: main.bib
title: Lattice-Based Unsupervised Test-Time Adaptation
  of Neural Network Acoustic Models

**Index Terms**: ASR, model adaptation, LF-MMI, LHUC

# Introduction

Acoustic model adaptation aims to improve automatic speech recognition (ASR) accuracy by reducing the mismatch between training and test conditions. In feature-space adaptation, transformations of acoustic features are estimated to maximise the log-likelihood of the adaptation data . A subset of the weights of a neural network acoustic model are adapted in model-based adaptation. Hybrid adaptation uses auxiliary features such as i-vectors  or speaker codes  to inform the acoustic model about speaker identities. Experiments have shown that these approaches are complementary and can be usefully combined .

A label sequence is provided for the adaptation data in supervised adaptation, but for unsupervised adaptation only an unlabelled recording is available. Conventionally, the best path from a first pass decoding is used to estimate labels for unsupervised adaptation . In this paper we focus on unsupervised test-time adaptation of neural network acoustic models.

An important challenge for unsupervised model adaptation of neural networks is that we do not want to overfit to errors made in the first pass decoding. In the past this challenge was tackled by filtering adaptation data by confidences produced by an ASR system  or by using an ASR quality estimation . Alternatively, neural network based acoustic models were prevented from overfitting to those errors by limiting expressivity of the adaptation by drastically reducing the number of adapted parameters, for example by adapting only amplitudes of hidden units  or by using low rank linear transformations . Furthermore, strong regularisers were used to prevent the outputs or weights of the acoustic model from diverging too far from the original model . In this paper we explore an alternative approach in which we use a lattice obtained from the first pass decoding as the supervision for unsupervised model adaptation, since lattices contain all information about the ASR system's confidence and possible phone confusions that can be leveraged during adaptation of the acoustic model. Lattices were previously used as supervision for unsupervised adaptation of Gaussian Mixture Models (GMM) using maximum likelihood linear regression (MLLR) for unsupervised training of GMMs using maximum likelihood training . However, in this paper we are interested in unsupervised adaptation of much larger discriminative models.

An effective unsupervised adaptation technique for neural network acoustic models requires three components. First, it is necessary to select a suitable subset of model parameters for adaptation in order to allow rapid adaptation using small amounts of adaptation data. Second, the system should filter the possible adaptation data with respect to its suitability for adaptation, as the first pass decoding may produce erroneous transcripts. Third, it needs a reliable adaptation schedule that updates the selected adaptation parameters using data filtered by the second component, while preventing overfitting to the adaptation data.

In this paper we explore an alternative solution to the data filtering component, in which all the adaptation data is used to adapt the whole neural network acoustic model, but the uncertainty in the decoding is captured through the use of complete lattices for supervision. Our approach is inspired by recent work on semi-supervised learning using the sequence level lattice-free maximum mutual information (LF-MMI) objective , in which it is shown that using lattices as supervision is beneficial compared to using only best paths in the semi-supervised learning setting. We acknowledge that semi-supervised training and test-time adaptation are essentially equal, but we emphasise that this approach allows us to reliably adapt all weights of neural network models using unsupervised adaptation and a discriminative training criterion, which was problematic in the past. Moreover, test-time speaker adaptation uses much less data than semi-supervised training (from 5 minutes to 1 hour) and it uses the same data for adaptation and testing.

We compare the lattice approach to using the best path, both obtained from the first pass decoding for unsupervised adaptation using the LF-MMI framework. This is experimentally explored using three transcription tasks – TED talks , multi-genre TV broadcasts , and a low-resource language, Somali – which have a wide range of baseline word error rates (WERs) (from 10% to 57%). We demonstrate improvements compared to using only the best path as supervision. Moreover, we show that by using this approach adapting all the parameters achieves better results than commonly used methods that adapt only a small subset of the weights, such as LHUC .

# Methods

## Lattice supervision and LF-MMI

Discriminative training using criteria such as maximum mutual information (MMI) has been shown to be sensitive to the accuracy of the transcripts . In lieu of better transcripts, a range of transcript filtering approaches have previously been explored . In unsupervised or semi-supervised approaches, in which we generate hypothesis transcriptions by decoding with a seed model, we can alternatively use a lattice of supervision. For instance, with the MMI criterion:

$$\mathcal{F}_{MMI}(\lambda) = \sum_{r=1}^R \log \frac{p_\lambda (\mathcal{O}_r | \mathcal{M}_r^{num})}{p_\lambda (\mathcal{O} | \mathcal{M}_r^{den})},$$ the numerator lattice $\mathcal{M}^{num}$ can contain multiple hypotheses for the same audio segment $r$ up to some lattice pruning factor. If set to 0, $\mathcal{M}^{num}$ is left with only the best path.

Lattice supervision has previously been used in work on unsupervised adaptation  and training  of GMMs, as well as discriminative and semi-supervised training of neural network models. Following Manohar et al. , we explore the use of lattice supervision versus that of only using the best path in the denominator lattice-free version of MMI (LF-MMI) . LF-MMI was introduced by Povey et al. as a method to train neural network acoustic models with a sequence discriminative criterion (MMI) without an initial cross-entropy (CE) stage to generate lattices approximating all possible word sequences (e.g. ). The word-level denominator lattice is instead replaced with a phone-level denominator graph encoding all possible sequences given a 3 or 4-gram phone language model. To further reduce complexity, the model outputs at one third of the frame rate. In the numerator a frame-by-frame mask allows phones to appear with some tolerance relative to its original alignment. A mixture of regularisation methods are required to reduce overfitting; for more details we refer to . Povey et al. demonstrated up to 8% relative improvements in WER over previous CE systems followed by sequence discriminative training with the state Minimum Bayes Risk (sMBR) criterion . An extension to the LF-MMI framework was recently proposed that enables flat-start training with neural networks .

## LHUC

Traditionally, only a subset of the acoustic model weights is adapted to prevent overfitting to the transcripts obtained from a first pass decode. This is usually done by inserting a linear layer after the input layer, hidden layers or output layers. For example, an adaption of activations $h$ of a hidden layer with speaker dependent weights $A$, can be expressed as follows:

$$h' = A \cdot h.$$

However, even these simple techniques tend to overfit because too many parameters are used for adaptation. Learning Hidden Unit Contributions (LHUC) is a technique in which only elements on the diagonal of the speaker dependent matrix $A$ are adapted – i.e. each hidden unit maybe viewed as having a speaker adaptive amplitude parameter. Since a much smaller number of weights is adapted, this technique is not that prone to overfitting to adaptation data.

LHUC was developed in the context of frame-based hybrid neural network / hidden Markov models, trained using the cross-entropy criterion. It has previously been successfully applied to sequence discriminatively trained models by using cross-entropy for the LHUC updates .

## SAT-LHUC

A way to further improve model adaptation is to use speaker adaptive training . Therefore, we also trained a SAT-LHUC model . This was done by maintaining speaker-dependent LHUC parameters for each speaker and training them jointly with speaker-independent parameters. In order to obtain a good speaker-independent model that can be used for decoding and adaptation to new speakers, we trained speaker-independent LHUC parameters instead of speaker dependent with probability 0.5 during training. This is similar to a setup that was shown to well in . At the beginning of training all speaker dependent parameters were sampled from a normal distribution with $\mu=1$ and $\sigma=0.01$. It was important to turn off L2 regularisation and parameter shrinkage for the speaker dependent parameters in Kaldi, as otherwise all speaker dependent parameters would converge to zero.

# Experiments

We conducted test-time model adaptation experiments on three datasets: the TED-LIUM corpus of TED talks , multi-genre TV broadcasts from the MGB 1 Challenge  and a corpus of Somali from the IARPA MATERIAL programme. All models were trained and adapted using the Kaldi toolkit . We describe the respective baseline models in sections 3.1-3.3 and the adaptation of the models in 3.4.

## TED-LIUM

We trained three time-delayed neural network (TDNN) models  with LF-MMI  following Kaldi TED-LIUM recipe 1f. All models had the same architecture with 7 hidden layers with 450 units. The first model was trained without i-vector features, the second model was trained with i-vector features and the third model was trained without i-vector features but with SAT-LHUC . All models were trained only on TED talks that were recorded before 2012 in order to conform with the IWSLT  evaluation guidelines which resulted in 130 hours of training data. We performed adaptation on the TED-LIUM dev and test data which contained 8 speakers with an average speech duration of 11.9 minutes and 11 speakers with an average speech duration of 14.2 minutes respectively.

## MGB

For the MGB-1 corpus we trained a factored TDNN (TDNN-F) model  with LF-MMI following Kaldi Switchboard recipe 7p. The model has 12 layers with 1280 units each (apart from the penultimate layer) and a bottleneck dimension of 256, with batch normalisation and dropout layers interleaved throughout. The model was trained for 8 epochs. We used alignments obtained with the HMM-GMM recipe provided with the MGB challenge, and trained on transcripts from a lightly supervised decode that had a maximum matching error rate (MER) with the original subtitles of 40. This yields roughly 649 hours of data, or 1960 hours after speed perturbation. For adaptation we carry across all training parameters (including dropout which we found particularly important for this data) and we rescore the supervision lattices with a 4-gram language model (LM) as in , estimated on about 640 million words of BBC subtitle text. We adapt and test on the longitudinal eval set which consists of 10 hours across two TV shows and a total of 19 episodes, each between 30 and 45 minutes in length. We do not have speaker clustering and therefore extract i-vectors per utterance and perform episode level adaptation. Finally, we rescore the decoded output with the 4-gram LM.

## Somali

We carried out experiments on Somali "surprise language" data released to participants on the IARPA-MATERIAL programme[^1]. Training data comprises 499 narrow-band telephone conversations sides, totalling 37 hours of speech. Test data comprises narrowband telephone conversations (NB); and wideband (WB) data from the news and topical broadcast domains that are mismatched to the training material. We trained a TDNN-F model using the neural network architecture from Kaldi TED-LIUM recipe 1g. The model had 14 hidden layers with 1024 units. The weight matrices were factored into two matrices with a bottleneck dimension 128. The model used filterbank, pitch and probability of voicing  features together with multilingual bottleneck features obtained from a neural network that was trained on all Babel languages . We used per utterance cepstral mean and variance normalisation, since there were no speaker clusters for the wideband test data.

The model was trained on narrowband data with speed perturbation and evaluated on both narrowband and wideband data. We used data scraped from the web to build a language model for wideband data. We performed speaker adaptation on narrowband data which consisted of 117 speakers with an average speech duration of 4.7 minutes and file-level adaptation on wideband data which consisted of 119 files with an average speech duration of 5 minutes.

## Adaptation methods

In this paper we were primarily interested in comparing model adaptation methods that use either one best path (called **BP** in the Results section) or a lattice (called **LAT** in the Results section) obtained from the first pass decoding for supervision. We adjusted a recipe for semi-supervised training using LF-MMI  to instead perform test-time adaptation. Our main hypothesis was that methods using lattices for supervision are much less likely to overfit to incorrectly transcribed segments in the adaptation data. In the past when only the best path was used for model adaptation, several techniques for data selection were required . In this paper we compared adapting using only utterances with top $25\%$, $50\%$ or $75\%$ average utterance confidence. We conducted model adaptation experiments in two regimes: in the first regime, we adapted all parameters of the acoustic model (called **ALL** in the Results section); in the second regime, we adapted only LHUC parameters inserted after every hidden layer of the acoustic model (called **LHUC** in the Results section). When adapting all parameters, we adapted the model for three epochs, starting with the learning rate which was used in the last iteration during training. We gradually decreased the learning rate down to one tenth (one fifth for MGB) of the initial learning rate. This learning schedule was chosen in order to imitate continued learning of the model. When adapting LHUC parameters, we adapted the model for three epochs with a fixed learning rate of $0.7$, which we found to work well in previous experiments.

# Results

We conducted the first set of experiments on the TED-LIUM dataset. Adaptation of the model without i-vectors using lattices achieves $10-15\%$ relative improvement when adapting LHUC parameters and $9-14\%$ relative improvement when adapting all parameters, whereas improvements when adapting using best path and all adaptation data were much smaller or even negative (Table ). We observed a similar trend for the remaining two models. Adaptation of the model using i-vectors (Table ) using lattices as supervision improves performance of a speaker adaptive baseline. This confirms that model-based adaptation is complementary with i-vectors. Unfortunately, our implementation of SAT-LHUC (Table ) did not outperform test-only LHUC adaptation. We plan to explore other possibilities of SAT-LHUC training in the future.

For the MGB corpus we adapted to entire episodes in the longitudinal eval set, rather than to speakers, as noted in Section . This provides more adaptation data (30-45 minutes per episode), but perhaps at the cost of losing finer granularity for adaptation. Adapting all parameters with lattice supervision provided the best results (Table ). This is to our knowledge the best results shown on the longitudinal evaluation set . Using the best path with all parameters yields almost no gains ($\sim1\%$). When only adapting a subset of the parameters with LHUC the results are more stable, but does not perform as well as all parameters with lattice supervision.

We also evaluated adaptation using lattices as supervision on the Somali data. As can be seen from the table, Somali data is very challenging – the initial WER of the model is very high on both NB and WB data at $53.7\%$ and $57.3\%$ respectively. These results are similar to other experiments conducted on other IARPA-MATERIAL programme languages with the same TDNN-F neural network architecture . Here we show that adapting such a model using a best path as supervision does not reduce the WER, because the best path contains too many errors. Nevertheless, adaptation using lattices as supervision gives $0.7-0.8\%$ absolute improvements. Even though the relative improvement is small, it is interesting to see that using lattices as supervision allows us to improve performance at all. We believe that adapting to entire files is sub-optimal, because the speaker variance in the wide-band data might be too high. Therefore, we plan to perform per utterance adaptation experiments in the future.

One common way to prevent adaptation to erroneous first pass transcripts is to filter the adaptation data by confidences , for example by the average utterance confidence. This filtering can be done by using a hard threshold, or by using only the fraction of utterances with the highest confidences. Either way one extra hyper-parameter is introduced. In Table  we compare adaptation using lattices as supervision with adaptation using only best paths on various fractions of the adaptation data when adapting all parameters. We experiment with the TED-LIUM model without i-vectors, and the Somali model. As can be seen from the table, filtering utterances improves results when using best path supervision. The biggest improvement can be achieved when using only $50\%$ of the adaptation data. Even then the TED-LIUM model does not obtain similar performance as when adapted using lattices for supervision. Furthermore, adaptation of the Somali model using best path supervision only barely matches the unadapted baseline. This is probably due to the fact that the WER of the initial Somali model is high and that the lattice provides much more information than a combination of best path supervision and corresponding confidences. We also performed the same filtering experiment with lattices as supervision. We found that using a threshold of 75% – 100% achieves the best results. Overall, adaptation using lattice supervision does not benefit from filtering utterances as much as adaptation using best path supervision.

# Conclusions

In this paper we compared unsupervised model adaptation using a lattice with the best path obtained from the first pass decoding as supervision. Our experiments show that using the lattice as supervision achieves better results than using the best path, even when confidence-based data selection is used to remove transcripts with many possible errors. This is due to the fact that the lattice from the first pass decoding contains much more information, such as confidence and phonetic confusions, than the best path. We find that the use of a lattice as supervision is particularly important when adapting all parameters, when over-fitting to incorrect first-pass transcriptions is a particular problem: in many cases we outperform a strong baseline that adapts only LHUC parameters. Moreover, we showed that when using lattices as supervision it is possible to adapt a model whose initial WER is higher than $50\%$, for which adapting with best path supervision often produced worse WERs than the unadapted baseline.

Our finding that use of lattices greatly aids the two adaptation methods we considered motivates further investigation into whether other test-time adaptation techniques – many of which show limited gains in an unsupervised setting – could benefit similarly. This will be the subject of further work.

**Acknowledgements:** This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Air Force Research Laboratory (AFRL) contract \#FA8650-17-C-9117. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, AFRL or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. This work was also partially supported by: the EU H2020 projects SUMMA (grant agreement 688139) and ELG (grant agreement 825627), and a PhD studentship funded by Bloomberg.

[^1]: https://www.iarpa.gov/index.php/research-programs/material


abstract: Such modern applications of topology as digital image analysis and data analysis have to deal with noise and other uncertainty. In this environment, the data structures often appear "filtered" into a sequence of cell complexes. We introduce the homology group of the filtration as the group of all possible homology classes of all elements of the filtration without double count. The second step of analysis is to discard the features that lie outside the user's choice of the acceptable level of noise.
address: Department of Mathematics, Marshall University, One John Marshall Drive, Huntington, WV 25755
author: Peter Saveliev
title: Robustness of topology of digital images and point clouds

# Introduction

Since Poincaré, homology has been used as the main descriptor of the topology of geometric objects. In the classical context, however, all homology classes receive equal attention. Meanwhile, applications of topology in analysis of images and data have to deal with noise and other uncertainty. This uncertainty appears usually in the form of a real valued function defined on the topological space. Persistence is a measure of robustness of the homology classes of the lower level sets of this function , , , .

Since it's unknown beforehand what is or is not noise in the dataset, we need to capture all homology classes including those that may be deemed noise later. In this paper we introduce an algebraic structure that contains, without duplication, all these classes. Each of them is associated with its persistence and can be removed when the acceptable threshold for noise is set. The last step can be carried out repeatedly in order to find the best possible threshold. The approach follows the approach to analysis of digital images presented in .

# Backgound

The topological spaces subject to such analysis are cell complexes. A *cell complex* is a combinatorial structure that describes how $k$ -dimensional cells are attached to each other along $(k-1)$-dimensional cells. Cell complexes come from the following two main sources.

First, a gray scale image is a real-valued function $f$ defined on a rectangle. Given a threshold $r$, the lower level set $f^{-1}((-\infty ,r))$ can be thought of as a binary image. Each black pixel of this image is treated as a square cell in the plane. These 2-dimensional cells are combined with their edges (1-cells) and vertices (0-cells) and in the $n$ -dimensional case, the image is decomposed into a combination of $0$-, $1$-, ..., $n$-cubes. This process is called *thresholding*. The result is a cell complex $K$ for each $r,$ see .

Second, a point cloud is a finite set $S$ in some Euclidean space of dimension $d$. Given a threshold $r$, we deem any two points that lie within $r$ from each other as "close". In this case, this pair of points is connected by an edge. Further, if three points are "close", pairwise, to each other, we add a face spanned by these points. If there are four, we add a tetrahedron, and, finally, any $d+1$ "close" points create a $d$-cell. The process is called the *Vietoris-Rips construction*. The result is a cell complex $K$ for each $r$ .

Next, we would like to quantify the topology of the cell complex $K.$ It is done via the *Betti numbersof* $K$: $B_{0}$ is the number of connected components in $K$; $B_{1}$ is the number of holes or tunnels (1 for letter O or the donut; 2 for letter B and the torus); $B_{2}$ is the number of voids or cavities (1 for both the sphere and the torus), etc.

The Betti numbers are computed via *homology theory* . One starts by considering the collection * *$C_{k}(K)$ of all formal linear combinations (over a ring $R$) of $k-$cells in $K,$ called *chains*. Combined they form a finitely generated abelian group called the *chain complex* $C_{k}(K)$, or collectively $C_{\ast }(K).$ A $k$ -chain can be recorded as an $N_{k}$-vector, where $N_{k}$ is the total number of $k$-cells in $K$. The boundary of a $k$-chain is the chain comprised of all $(k-1)$-faces of its cells taken with appropriate signs. Then the *boundary operator* $\partial :C_{k}(K)\rightarrow
C_{k-1}(K)$ acts on the chain complex and is represented by a $N_{k}\times
N_{k-1}$ matrix.

From the chain complex $C_{\ast }(K),$ the homology group is constructed by means of the standard algebraic tools. To capture the topological features one concentrates on *cycles*, i.e., chains with zero boundary, $%
\partial A=0$. Further, one can verify whether two given $k$-cycles $A$ and $%
B$ are *homologous*: the difference between them is the boundary of a $(k+1)$-chain $T:A-B=\partial T$ (such as two meridians of the torus). In this case, $A$ and $B$ belong to the same *homology class* $H=[A]=[B]$ . The totality of these equivalence classes in each dimension $k$ is called the $k$-th *homology group* $H_{k}(K)$ of $K$, collectively $H_{\ast
}(K)$. Then, Betti number $B_{k}$ is the rank of $H_{k}(K)$.

# Prior work and outline

The methods for computing homology groups are well developed. In real-life applications however both digital images and point clouds may be noisy and one needs to evaluate the significance of their homology classes. The approach to this problem has been the following. Instead of using a single threshold and studying a single cell complex, one considers all thresholds and all possible cell complexes. Since increasing threshold $r$ enlarges the corresponding complex, we have a sequence of complexes: $$K^{1}\hookrightarrow K^{2}\hookrightarrow K^{3}\hookrightarrow
K^{4}\hookrightarrow \ldots \ \hookrightarrow K^{s},$$ where the arrows represent the inclusions: $i^{n,n+1}:K^{n}\hookrightarrow
K^{n+1}.$ Let $i^{nm}:K^{n}\hookrightarrow K^{m},n\leq m,$ also be the inclusion. This structure $\{K^{n},i^{nm}\}$ is called a *filtration*$%
.$

Now, each of these inclusions generates a homomorphism $i_{\ast
}^{nm}:H_{\ast }(K^{n})\rightarrow H_{\ast }(K^{m})$ called the *homology map induced by* $i^{nm}.$ As a result, we have a sequence of homology groups connected by these homomorphisms: $$H_{\ast }(K^{1})\rightarrow H_{\ast }(K^{2})\rightarrow \ldots \ \rightarrow
H_{\ast }(K^{s})\longrightarrow 0.$$ These homomorphisms record how the homology changes as the complex grows at each step. For example, a component appears and then merges with another one, or a hole is formed and then filled. We refer to these events as *birth and death* of the corresponding homology classes.

In order to evaluate the robustness of an element of one of these groups the *persistence of a homology class* is defined as the number of steps it takes for the class to end at $0.$ In other words, $$\text{persistence = death date - birth date.}$$ The $p$-*persistent homology group* of $K^{i}$ is defined as the image of $i_{\ast }^{i,i+p}.$ It's what's left from $H_{\ast }(K^{i})$ after $p$ steps in the filtration. Now the robustness of the homology classes of the filtration is evaluated in terms of the set of intervals $[birth,death]$ representing the life-spans, called *barcodes*, of the homology classes .

Our approach is somewhat different. It consists of two steps.

First step: we pool all possible homology classes in all elements of the filtration together in a single algebraic structure (Sections 4 and 5). The presence of noise is ignored. The homology group $H_{\ast }(\{K^{n}\})$ of filtration* *$\{K^{n}\}$ captures all homology classes in the whole filtration – without double counting.

Second step: for a given positive integer $p,$ the $p$*-*noise group * *$N_{\ast }^{p}(\{K^{n}\})$ is comprised of the homology classes in $H_{\ast }(\{K^{n}\})$ with the persistence less than $p.$ Next, we "remove" the noise from the homology group of filtration by using the quotient (Sections 6 and 7): $$H_{\ast }^{p}(\{K^{n}\})=H_{\ast }(\{K^{n}\})/N_{\ast }^{p}(\{K^{n}\}).$$ In other words: *if the difference between two homology classes is deemed noise, they are equivalent*. The second step can be repeated as needed.

We also discuss the computational aspects of this approach (Section 8) and multiparameter filtrations (Section 9).

Our approach provides a coarser classification of the homology of filtrations than the one based on barcodes. The reason is that all homology classes with long enough life-spans, i.e., high persistence, have equal place in the homology group $H_{\ast }(\{K^{n}\})$ of the filtration regardless of the time of birth and death.

# Motivation: the homology of a gray scale image

In this section we will try to understand the meaning of the homology of the gray scale image in Figure 1. For simplicity we assume that there are only 2 levels of gray in addition to black and white. A visual inspection of the image suggests that it has three connected components each with a hole. Therefore, its $0$- and $1$-homology groups shouldhave three generators each. We now develop an algebraic procedure to arrive at this result.

First the image is "thresholded". The lower level sets of the gray scale function of the image form a filtration: a sequence of three binary images, i.e. cell complexes: $K^{1}\hookrightarrow K^{2}\hookrightarrow K^{3},$ where the arrows represent the inclusions. Suppose $A_{i},B_{i},C_{i}$ are the homology classes that represent the components of $K^{i}$ and $%
a_{i},b_{i},c_{i}$ are the holes, clockwise starting at the upper left corner. The homology groups of these images also form sequences – one for each dimension 0 and 1.

Suppose $F_{1},F_{2}$ are the two homology maps, i.e., homomorphisms of the homology groups generated by the inclusions of the complexes, with $F_{3}=0$ included for convenience. These homomorphisms act on the generators, as follows: $$\begin{aligned}
A_{1} &\rightarrow &A_{2}\rightarrow A_{3}\rightarrow 0,B_{1}\rightarrow
B_{2}\rightarrow B_{3}\rightarrow 0, \\
C_{2} &\rightarrow &C_{3}\rightarrow 0,a_{1}\rightarrow a_{2}\rightarrow
a_{3}\rightarrow 0, \\
b_{1} &\rightarrow &0,c_{3}\rightarrow 0.
\end{aligned}$$ To avoid double counting, we want to count only the homology classes that don't reappear in the next homology group. As it turns out, a more algebraically convenient way to accomplish this is to count only the homology classes that go to $0$ under these homomorphisms. These classes form the kernels of $F_{1},F_{2},F_{3}$. Now, we choose the homology group of the original, gray scale image to be the direct sum of these kernels: $$H_{0}(\{K^{i}\})=<A_{3},B_{3},C_{3}>,\text{ }H_{1}(\{K^{i}%
\})=<b_{1},a_{3},c_{3}>.$$ Thus the image has three components and three holes, as expected.

# Homology groups of filtrations

In the following sections we provide formal definitions. All cell complexes are finite.

Suppose we have a one-parameter filtration: $$K^{1}\hookrightarrow K^{2}\hookrightarrow K^{3}\hookrightarrow \ldots \
\hookrightarrow K^{s}.$$ Here $K^{1},K^{2},\ldots ,K^{s}$  are cell complexes and the arrows represent the inclusions $i^{n,n+1}:K^{n}\hookrightarrow K^{n+1}$ and so do $%
i^{nm}:K^{n}\hookrightarrow K^{m},n\leq m$. We will denote the filtration by $\{K^{n},i^{nm}:n,m=1,2,...,s,n\leq m\},$ or simply $\{K^{n}\}.$ Next, homology generates a "direct system" of groups and homomorphisms: $$H_{\ast }(K^{1})\rightarrow H_{\ast }(K^{2})\rightarrow \ldots \ \rightarrow
H_{\ast }(K^{s})\longrightarrow 0.$$ We denote this direct system by $\{H_{\ast }(K^{n}),i_{\ast
}^{nm}:n,m=1,2,...,s,n\leq m\},$ or simply $\{H_{\ast }(K^{n})\}.$ The zero is added in the end for convenience.

Our goal is to define a single structure that captures all homology classes in the whole filtration without double counting. The rationale is that if $%
x\in H_{\ast }(K^{n}),y\in H_{\ast }(K^{m}),$ $y=i_{\ast }^{nm}(x),$ and there is no other $x$ satisfying this condition, then $x$ and $y$ may be thought of as representing the same homology class of the geometric object behind the filtration.

The *homology group of filtration* $\{K^{n}\}$ is defined as the product of the kernels of the inclusions: $$H_{\ast }(\{K^{n}\})=\ker i_{\ast }^{1,2}\oplus \ker \,i_{\ast }^{2,3}\oplus
\ldots \oplus \ker i_{\ast }^{s,s+1}.$$ Here, from each group we take only the elements that are about to die. Since each dies only once, there is no double-counting. Since the sequence ends with $0,$ we know that everyone will die eventually. Hence every homology class appears once and only once.

These are a few simple facts about this group.

**Proposition 1**. *If $i_{\ast}^{n,n+1}$ is an isomorphism for each $n=1,2,...,s-1,$ then $H_{\ast}(\{K^{n}\})=H_{\ast}(K^{1})$ $.$*

**Proposition 2**. *If $i_{\ast}^{n,n+1}$ is a monomorphism for each $n=1,2,...,s-1,$ then $H_{\ast}(\{K^{n}\})=H_{\ast}(K^{s}).$*

**Proposition 3**. *Suppose $\{K^{n},i^{nm},n,m=1,2,...,s\}$ and $\{L^{n},j^{nm},n,m=1,2,...,s\}$ are filtrations. Then $H_{\ast}(\{K^{n}\sqcup L^{n}\})=H_{\ast}(\{K^{n}%
\})\oplus H_{\ast}(\{L^{n}\}).$*

**Proposition 4**. *Suppose $\{K^{n},i^{nm},n,m=1,2,...,s\}$ and $\{L^{n},j^{nm},n,m=1,2,...,s\}$ are filtrations and $f:K^{s}\rightarrow L^{s}$ is a cell map. Then the homology map of the homology groups of these filtrations $f_{\ast}:H_{\ast
}(\{K^{n}\})\rightarrow H_{\ast}(\{L^{n}\})$ is well defined as $$f_{\ast}(x_{1},x_{2},...,x_{s})=(f_{\ast}^{1}(x_{1}),f_{\ast}^{2}%
(x_{2}),...,f_{\ast}^{s}(x_{s})),$$ where $f^{n}$ is the restriction of $f$ to $K^{n}.$*

The stability of the homology group of a filtration follows from the stability of its persistence diagram, i.e., the set of points $%
\{(birth,death)\}\subset \mathbf{R}^{2}$ for the generators of the homology groups of the filtration, plus the diagonal. It is proven in that $d_{B}(D(f),D(g))\leq ||f-g||_{\infty },$ where $d_{B}$ is the bottle-neck distance between the persistence diagrams $D(f),D(g)$ of two filtrations generated by tame functions $f,g.$ Function $F(x,y)=y-x$ creates an analogue bottle-neck distance for the set of points $\{persistence\}\subset \mathbf{R}$ and its stability follows from the continuity of $F$.

# Motivation: the high contrast homology of a gray scale image

To justify our approach to persistence, we observe that some of the features in the image in Figure 1 are more prominent than others. In particular, some of the features have lower contrast. These are the holes in the second and the third rings as well as the third ring itself. By *contrast* of a lower level set of the gray level function we understand the difference between the highest gray level adjacent to the set and the lowest gray level within the set.

An easy computation shows that the homology classes with persistence of 3 or higher among the generators are: $A_{1},B_{1},a_{1}.$ However, the set of the classes of high persistence isn't a subgroup of the homology group of the respective complex. Instead, we look at the classes with *low* persistence, i.e., the noise. In particular, the classes in $H_{\ast }(K^{1})$ of persistence 2 or lower form the kernel of $F_{2}F_{1}$. We now "remove" this noise from the homology groups of the filtration by considering their quotients over these kernels. In particular, the 3-persistent homology groups of the image are: $$\begin{aligned}
H_{0}^{3}(\{K^{i}\})& =<A_{1},B_{1}>/0=<A_{1},B_{1}>, \\
H_{1}^{3}(\{K^{i}\})& =<a_{1},b_{1}>/<b_{1}>=<a_{1}>.
\end{aligned}$$ Observe that the output is identical to the homology of a single complex, i.e., a binary image, with two components and one hole. The way persistence is defined ensures that we can never remove a component as noise but keep a hole in it.

Observe now that the holes in the second and third rings have the same persistence (contrast) and, therefore, occupy the same position in the homology group regardless of their birth dates (gray level). Second, if we shrunk one of these rings, its persistence and, therefore, its place in the homology group wouldn't change. These observations confirm the fact that the homology group of the gray scale image, unlike the barcodes, captures only its topology.

In the case of a Vietoris-Ripps complex, not only the barcode, \[birth, death\], but also the persistence, death - birth, of a homology class contains information about the size of representatives of these classes. For example, a set of points arranged in a circle will produce a 1-cycle with twice as large birth, death, and persistence than the same set shrunk by a factor of 2. However, persistence defined as death/birth will have the desired property of scale independence. The same result can be achieved by an appropriate re-parametrizing of the filtration.

# Persistent homology groups of filtrations

In the general context of filtrations the measure of importance of a homology class is its persistence which is the length of its lifespan within the direct system of homology of the filtration.

Given filtration $\{K^{n}\},$ we say that *the persistence* $P(x)$ *of* $x\in H_{\ast}(K^{n})$* is equal to* $p$ if $i_{\ast
}^{n,n+p}(x)=0$ and $i_{\ast}^{n,n+p-1}(x)\neq0.$ Our interest is in the "robust" homology classes, i.e. the ones with high persistence. However, the collection of these classes is not a group as it doesn't even contain 0. So we deal with "noise" first. Given a positive integer $p,$ the $p$ *-noise (homology) group* $N_{\ast}^{p}(K^{n})$ of $\{K^{n}\}$ is the group of all elements of $K^{n}$ with persistence less than $p.$

Alternatively, we can define these groups via kernels of the homomorphisms of the inclusions: $N_{\ast }^{p}(K^{n})=\ker \,i_{\ast }^{n,n+p}.$

**Proposition 5**. *$N_{\ast}^{p+1}(K^{n})\subset N_{\ast}^{p}(K^{n}).$*

Next, we "remove" the noise from the homology group. The $p$ *-persistent (homology) group* of $K^{n}$ with respect to the filtration $%
\{K^{n}\}$ is defined as $$H_{\ast }^{p}(K^{n})=H_{\ast }(K^{n})/N_{\ast }^{p}(K^{n}).$$ The point of this definition is that, given a threshold for noise, if the difference between two homology classes is noise, they should be equivalent.

Next, just as in the case of noise-less analysis, we define a single structure to capture all (robust) homology classes. Let $p$ be a positive integer$.$ Suppose $x\in \ker i_{\ast }^{k,k+p}$ and let $y=$ $i_{\ast
}^{k,k+1}(x).$ Then $$\begin{aligned}
i_{\ast }^{k+1,k+1+p}(y) &=&i_{\ast }^{k+1,k+1+p}(i_{\ast }^{k,k+1}(x)) \\
&=&i_{\ast }^{k,k+1+p}(x)=i_{\ast }^{k+p,k+p+1}(i_{\ast }^{k,k+p}(x)) \\
&=&i_{\ast }^{k+p,k+p+1}(0)=0.
\end{aligned}$$ Hence $y\in \ker i_{\ast }^{k+1,k+1+p}.$ We have proved that $$i_{\ast }^{k,k+1}(\ker i_{\ast }^{k,k+p})\subset \ker i_{\ast }^{k+1,k+1+p}.$$ It follows that the homomorphism $i_{\ast }^{k,k+1}:\ker i_{\ast
}^{k,k+p}\rightarrow \ker i_{\ast }^{k+1,k+1+p}$ generated by the inclusion is well-defined.

Next, we use these homomorphisms to define the $p$*-noise (homology) group* $N_{\ast }^{p}(\{K^{n}\})$* of filtration* $\{K^{n}\}$ as $$N_{\ast }^{p}(\{K^{n}\})=\ker i_{\ast }^{1,2}\oplus \ldots \oplus \ker
i_{\ast }^{s,s+1}.$$ Observe that the formula is the same as the one in the definiton of $H_{\ast
}^{p}(\{K^{n}\}).$ Since $i_{\ast }^{k,k+1}:\ker i_{\ast
}^{k,k+p}\rightarrow \ker i_{\ast }^{k+1,k+1+p}$ is a restriction of $%
i_{\ast }^{k,k+1}:H_{\ast }^{p}(K^{k})\rightarrow H_{\ast }^{p}(K^{k+1}),$ each term in the above definition is a subgroup of the corresponding term in the definition of $H_{\ast }(\{K^{n}\}).$ The proposition below follows.

**Proposition 6**. *$N_{\ast }^{p}(\{K^{n}\})\subset H_{\ast }(\{K^{n}\}).$*

Finally, the* *$p$*-persistent (homology) group of filtration* $\{K^{n}\}$ is $$H_{\ast }^{p}(\{K^{n}\})=H_{\ast }(\{K^{n}\})/N_{\ast }^{p}(\{K^{n}\}).$$

The results about $H_{\ast }^{p}(\{K^{n}\})$ analogous to the ones about $%
H_{\ast }(\{K^{n}\})$ in Section 5 hold.

# Computational aspects

For 2-dimensional gray scale images, this approach to homology and persistence has been used in an image analysis program. The algorithm described in has complexity of $O(n^{2}),$ where $n$ is the number of pixels in the image.

For the general case, the analysis algorithm may be outlined as follows:

1.  The input is a filtration.

2.  The homology groups of its members and the homomorphisms induced by inclusions are computed.

3.  The homology group of the filtration is computed.

4.  The persistence of all elements of the homology groups is computed.

5.  The user sets a threshold $p$ for persistence and the $p$-noise group of the filtration is computed.

6.  The $p$-persistent homology group of the filtration is computed and given as output.

If the user changes the threshold, the last step is repeated as necessary without repeating the rest.

The algorithm above computes the homology group of filtration, as defined, incrementally. This may be both a disadvantage and an advantage. In comparison, the persistence complex also contains information about all homology classes of the filtration but its computation does not require computing the homology of each complex of the filtration. Meanwhile, the above algorithm may have to compute the same homology over and over if consecutive complexes are identical. Hence, the algorithm has a disadvantage in terms of processing time. On the other hand, the incremental nature of the algorithm makes its use of memory independent from the length of the filtration. Another advantage is that multi-parameter filtrations are dealt with in the exact same manner (see next section).

The inefficiency of the above algorithm can be addressed with a proper algebraic tool. This tool is the mapping cone . Suppose, for simplicity, that our filtration has only two elements: $i:K^{1}%
\hookrightarrow K^{2}.$ The mapping cone is, in a sense, a combination of the kernel and the cokernel of $i_{\ast }$. It captures the difference between $K^{1}$ and $K^{2}$ on the chain level: everything in $C_{\ast
}(K^{1})$ is killed unless it also appears in $C_{\ast }(K^{2})$ under $%
i_{\ast }$. Then the algorithm is to construct the homology group from the chain complexes $C_{\ast }(K^{1}),C_{\ast }(K^{2})$ of the elements of the filtration and the chain map $i_{\ast }:C_{\ast }(K^{1})\rightarrow C_{\ast
}(K^{2}).$

# Multiparameter filtrations

Multiparameter filtrations come from the same main sources as one-parameter filtrations. First, color images are thresholded according to their three color channels. Second, point clouds are thresholded by the closeness of their points and, for example, the density of hte points.

Let limit our attention to the two-parameter case. A (finite) two-parameter filtrations $\{K^{nm}\}$ is a table of complexes connected by inclusions $$i(n,m,n+p,m+q):K^{nm}\rightarrow K^{n+p,m+q},p,q\geq 0,$$ These inclusions generate homomorphisms $$i_{\ast }(n,m,n+q,m+p):H_{\ast }(K^{nm})\rightarrow H_{\ast }(K^{n+q,m+p}),$$ with 0s added in the end of each row and each column. Define the *homology group of the filtration* $\{K^{nm}\}$ as $$\begin{aligned}
H_{\ast }(\{K^{nm}\}) = {\displaystyle\bigoplus\limits_{n}}\ker i_{\ast }(n,m,n+1,m)\cap \ker i_{\ast
}(n,m,n,m+1).
\end{aligned}$$ The analogues of the results in Section 5 hold.

There are many ways to define persistence in the multiparameter setting. For example, we can evaluate the robustness of a homology class $x\in H_{\ast
}(K^{nm})$ in terms of the pairs $(p,q)$ of positive integers satisfying $$i_{\ast }(n,m,n+p,m)(x)=0\text{ and }i_{\ast }(n,m,n,m+q)(x)=0.$$ Next, just as in Section 7, we restrict the homomorphisms generated by the inclusions to the homology classes of low persistence: $$\begin{aligned}
i_{\ast }(n,m,n+1,m) &:& \\
\ker i_{\ast }(n,m,n+p,m) &\rightarrow &\ker i_{\ast }(n+1,m,n+1+p,m), \\
i_{\ast }(n,m,n,m+1) &:& \\
\ker i_{\ast }(n,m,n,m+q) &\rightarrow &\ker i_{\ast }(n+1,m,n,m+1+q).
\end{aligned}$$ Then the $(p,q)$-*noise group of* $K^{nm}$ is defined via these homomorphisms: $$\begin{aligned}
N_{\ast }^{pq}(\{K^{nm}\})  = {\displaystyle\bigoplus\limits_{n}}\ker i_{\ast }(n,m,n+1,m)\cap \ker i_{\ast
}(n,m,n,m+1).
\end{aligned}$$ Finally, the $(p,q)$*-persistent (homology) group of filtration* $%
\{K^{nm}\}$ is defined as $$H_{\ast }^{pq}(\{K^{n}\})=H_{\ast }(\{K^{nm}\})/N_{\ast }^{pq}(\{K^{nm}\}).$$

The results about $H_{\ast }^{pq}(\{K^{nm}\})$ analogous to the ones about $%
H_{\ast }^{p}(\{K^{n}\})$ in Section 7 hold.

[Robustness of topology of digital images and point clouds](http://inperc.com/wiki/index.php?title=Homology_of_filtrations)