Noname manuscript No.

(will be inserted by the editor)

Adapting general-purpose speech recognition engine
output for domain-specific natural language question
answering
C. Anantaram · Sunil Kumar Kopparapu

arXiv:1710.06923v1 [cs.CL] 12 Oct 2017

the date of receipt and acceptance should be inserted later

Abstract Speech-based natural language question-answering interfaces to enter-

prise systems are gaining a lot of attention. General-purpose speech engines can be
integrated with NLP systems to provide such interfaces. Usually, general-purpose
speech engines are trained on large ‘general’ corpus. However, when such engines
are used for specific domains, they may not recognize domain-specific words well,
and may produce erroneous output. Further, the accent and the environmental
conditions in which the speaker speaks a sentence may induce the speech engine to inaccurately recognize certain words. The subsequent natural language
question-answering does not produce the requisite results as the question does not
accurately represent what the speaker intended. Thus, the speech engine’s output
may need to be adapted for a domain before further natural language processing is carried out. We present two mechanisms for such an adaptation, one based
on evolutionary development and the other based on machine learning, and show
how we can repair the speech-output to make the subsequent natural language
question-answering better.

1 Introduction

Speech-enabled natural-language question-answering interfaces to enterprise application systems, such as Incident-logging systems, Customer-support systems,
Marketing-opportunities systems, Sales data systems etc., are designed to allow
end-users to speak-out the problems/questions that they encounter and get automatic responses. The process of converting human spoken speech into text is
performed by an Automatic Speech Recognition (ASR) engine. While functional
examples of ASR with enterprise systems can be seen in day-to-day use, most of
C. Anantaram
TCS Innovation Labs - Delhi, ASF Insignia, Gwal Pahari, Gurgaon, India
E-mail: c.anantaram@tcs.com
Sunil Kumar Kopparapu
TCS Innovation Labs mar.kopparapu@tcs.com

Mumbai,

Yantra

Park,

Thane

(West)

E-mail:

sunilku-

2

Anantaram, Sunil

these work under constraints of a limited domain, and/or use of additional domainspecific cues to enhance the speech-to-text conversion process. Prior speech-andnatural language interfaces for such purposes have been rather restricted to either
Interactive Voice Recognition (IVR) technology, or have focused on building a
very specialized speech engine with domain specific terminology that recognizes
key-words in that domain through an extensively customized language model, and
trigger specific tasks in the enterprise application system. This makes the interface
extremely specialized, rather cumbersome and non-adaptable for other domains.
Further, every time a new enterprise application requires a speech and natural
language interface, one has to redevelop the entire interface again.
An alternative to domain-specific speech recognition engines has been to repurpose general-purpose speech recognition engines, such as Google Speech API,
IBM Watson Speech to text API which can be used across domains with natural language question answering systems. Such general-purpose automatic speech
engines (gp-ASR) are deep trained on very large general corpus using deep neural
network (DNN) techniques. The deep learnt acoustic and language models enhance the performance of a ASR. However, this comes with its own limitations.
For freely spoken natural language sentences, the typical recognition accuracy
achievable even for state-of-the-art speech recognition systems have been observed
to be about 60% to 90% in real-world environments (Lee et al., 2010). The recognition is worse if we consider factors such as domain-specific words, environmental
noise, variations in accent, poor ability to express on the part of the user, or inadequate speech and language resources from the domain to train such speech
recognition systems. The subsequent natural language processing, such as that in
a question answering system, of such erroneously and partially recognized text
becomes rather problematic, as the domain terms may be inaccurately recognized
or linguistic errors may creep into the sentence. It is, hence, important to improve
the accuracy of the ASR output text.
In this paper, we focus on the issues of using a readily available gp-ASR
and adapting its output for domain-specific natural language question answering
(Anantaram et al., 2015a). We present two mechanisms for adaptation, namely

(a) an evolutionary development based artificial development mechanism of adaptation (Evo-Devo), where we consider the output of ASR as a biological entity
that needs to adapt itself to the environment (in this case the enterprise domain) through a mechanism of repair and development of its genes and
(b) a machine learning based mechanism where we examine the closest set of
matches with trained examples and the number of adaptive transformations
that the ASR output needs to undergo in order to be categorized as an acceptable natural language input for question-answering.

We present the results of these two adaptation and gauge the usefulness of each
mechanism. The rest of the paper is organized as follows, in Section 2 we briefly
describe the work done in this area which motivates our contribution. The main
contribution of our work is captured in Section 3 and we show the performance of
our approach through experiments in Section 4. We conclude in Section 5.

Adapting general-purpose speech recognition engine for QA

3

2 Related Work

Most work on ASR error detection and correction has focused on using confidence measures, generally called the log-likelihood score, provided by the speech
recognition engine; the text with lower confidence is assumed to be incorrect and
subjected to correction. Such confidence based methods are useful only when we
have access to the internals of a speech recognition engine built for a specific domain. As mentioned earlier, use of domain-specific engine requires one to rebuild
the interface every time the domain is updated, or a new domain is introduced.
As mentioned earlier, our focus is to avoid rebuilding the interface each time the
domain changes by using an existing ASR. As such our method is specifically
a post-ASR system. A post-ASR system provides greater flexibility in terms of
absorbing domain variations and adapting the output of ASR in ways that are
not possible during training a domain-specific ASR system (Ringger and Allen,
1996).
Note that an erroneous ASR output text will lead to an equally (or more)
erroneous interpretation by the natural language question-answering system,
resulting in a poor performance of the overall QA system

Machine learning classifiers have been used in the past for the purpose of combining features to calculate a confidence score for error detection. Non-linguistic
and syntactic knowledge for detection of errors in ASR output, using a support
vector machine to combine non-linguistic features was proposed in (Shi, 2008) and
Naive Bayes classifier to combine confidence scores at a word and utterance level,
and differential scores of the alternative hypotheses was used in (Zhou et al., 2005)
Both (Shi, 2008) and (Zhou et al., 2005) rely on the availability of confidence scores
output by the ASR engine. A syllable-based noisy channel model combined with
higher level semantic knowledge for post recognition error correction, independent
of the internal confidence measures of the ASR engine is described in (Jeong et al.,
2004). In (López-Cózar and Callejas, 2008) the authors propose a method to correct errors in spoken dialogue systems. They consider several contexts to correct
the speech recognition output including learning a threshold during training to
decide when the correction must be carried out in the context of a dialogue system. They however use the confidence scores associated with the output text to
do the correction or not. The correction is carried using syntactic-semantic and
lexical models to decide whether a recognition result is correct.
In (Bassil and Semaan, 2012) the authors proposes a method to detect and
correct ASR output based on Microsoft N-Gram dataset. They use a contextsensitive error correction algorithm for selecting the best candidate for correction
using the Microsoft N-Gram dataset which contains real-world data and word
sequences extracted from the web which can mimic a comprehensive dictionary of
words having a large and all-inclusive vocabulary.
In (Jun and Lei, 2011) the authors assume the availability of pronunciation
primitive characters as the output of the ASR engine and then use domain-specific
named entities to establish the context, leading to the correction of the speech
recognition output. The patent (Amento et al., 2007) proposes a manual correction of the ASR output transcripts by providing visual display suggesting the
correctness of the text output by ASR. Similarly, (Harwath et al., 2014) propose
a re-ranking and classification strategy based on logistic regression model to esti-

4

Anantaram, Sunil

mate the probability for choosing word alternates to display to the user in their
framework of a tap-to-correct interface.
Our proposed machine learning based system is along the lines of (Jeong et al.,
2004) but with differences: (a) while they use a single feature (syllable count)
for training, we propose the use of multiple features for training the Naive Bayes
classifier, and (b) we do not perform any manual alignment between the ASR and
reference text – this is done using an edit distance based technique for sentence
alignment. Except for (Jeong et al., 2004) all reported work in this area make
use of features from the internals of the ASR engine for ASR text output error
detection.
We assume the use of a gp-ASR in the rest of the paper. Though we use
examples of natural language sentences in the form of queries or questions, it
should be noted that the description is applicable to any conversational natural
language sentence.

3 Domain adaptation of ASR output

3.1 Errors in ASR output
In this paper we focus on question answering interfaces to enterprise systems,
though our discussion is valid for any kind of natural language processing sentences that are not necessarily a query. For example, suppose we have a retail-sales
management system domain, then end-users would be able to query the system
through spoken natural language questions (S ) such as
(
S=

/What is the total sales of miscellaneous store
retailers from year two thousand ten to year two
thousand fifteen?/

A perfect ASR would take S as the input and produce (T ), namely,
(
T =

what is the total sales of miscellaneous store
retailers from year two thousand ten to year two
thousand fifteen

We consider the situation where a ASR takes such a sentence (S ) spoken by a
person as input, and outputs an inaccurately recognized text (T 0 ) sentence. In our
experiments, when the above question was spoken by a person and processed by
a popular ASR engine such as Google Speech API, the output text sentence was
(T 0 )

what is the total sales of miscellaneous storyteller
T0 =
from the year two thousand ten to two thousand fifteen

Namely
S −→ ASR −→ T 0

It should be noted that an inaccurate output by the ASR engine maybe the
result of various factors such as background noise, accent of the person speaking
the sentence, the speed at which he or she is speaking the sentence, domainspecific words that are not part of popular vocabulary etc. The subsequent natural

Adapting general-purpose speech recognition engine for QA

5

language question answering system cannot answer the above output sentence
from its retail sales data. Thus the question we tackle here is – how do we adapt
or repair the sentence (T 0 ) back to the original sentence (T ) as intended by the
speaker. Namely
T 0 −→ adaptation, repair −→ T

We present two mechanisms for adaptation or repair of the ASR output, namely
T 0 −→ T , in this paper: (a) an evolutionary development based artificial develop-

ment mechanism, and (b) a machine-learning mechanism.

3.2 Evo-Devo based Artificial Development mechanism of adaption
Our mechanism is motivated by Evolutionary Development (Evo-Devo) processes
in biology (Harding and Banzhaf, 2008; Anantaram et al., 2015b; Tufte, 2008)
to help adapt/repair the overall content accuracy of an ASR output (T 0 ) for a
domain. We give a very brief overview of Evo-Devo process in biological organisms
and discuss how this motivates our mechanism. In a biological organism, evodevo processes are activated when a new biological cell needs to be formed or an
injured cell needs to be repaired/replaced. During such cell formation or repair,
the basic genetic structure consisting of the genes of the organism are replicated
into the cell – the resultant set of ’genes in the cell’ is called the genotype of the
cell. Once this is done, the genotype of the cell is then specialized through various
developmental processes to form the appropriate cell for the specific purpose that
the cell is intended for, in order to factor-in the traits of the organism – called the
phenotype of the cell. For example, if a person has blue-eyes then a blue-eye cell is
produced, or if a person has brown skin then a brown-skin cell is produced. During
this process, environmental influence may also play a role in the cell’s development
and such influences are factored into the genotype-phenotype development process.
The field of Evo-Devo has influenced the field of Artificial Intelligence (AI) and a
new sub-field called Artificial Development (Art-Dev) has been created that tries
to apply Evo-Devo principles to find elegant solutions to adaptation and repair
problems in AI.
We take inspiration from the Evo-Devo biological process and suitably tailor
it to our research problem of repairing the ASR output (T 0 ). In our approach
we consider the erroneous ASR output text as the input for our method and
treat it as an ’injured biological cell’. We repair that ’injured cell’ through the
development of the partial gene present in the input sentence with respect to
the genes present in the domain. We assume that we have been provided with
the domain ontology describing the terms and relationships of the domain. In
our framework, we consider the domain ontology as the true ’genetic knowledge’
of that ’biological organism’. In such a scenario, the ’genetic repair’ becomes a
sequence of match-and-replace of words in the sentence with appropriate domain
ontology terms and relationships. Once this is done, the ’genotype-to-phenotype
repair’ is the repair of linguistic errors in the sentence after the ’genetic repair’.
The following sub-section describes our method in detail.

6

Anantaram, Sunil

3.2.1 Repair method

We assume that all the instances of the objects in the domain are stored in a
database associated with the enterprise system, and can be expressed in relational
form (such as [a R c]), for example ['INDUSTRY', 'has', 'PEAK SALES']. A relational database will store it as a set of tables and we treat the data in the database
as static facts of the domain. The ontology of the domain can then be generated
from this database. We assume that the data schema and the actual data in the
enterprise application forms a part of the domain terms and their relationships
in the ontology. This identifies the main concepts of the domain with a <subjectpredicate-object> structure for each of the concepts. The ontology thus generated
describes the relations between domain terms, for example ['SALES', 'has code',
'NAICS CODE'] or ['OPTICAL GOODS', 'sales 2009', '8767 million'] and thus can
be expressed using OWL schema as <s-p-o > structure. Each <s-p-o> entry forms
the genes of the domain.
We start by finding matches between domain ontology terms and words that
appear in the input sentence. Some words of the input sentence will match domain
ontology terms exactly. The corresponding domain ontology entry consisting of
subject-predicate-object triple is put into a candidate set. Next, other words in
the input sentence that are not exact matches of domain ontology terms but have
a ’closeness’ match with terms in the ontology are considered. This ’closeness’
match is performed through a mix of phonetic match combined with Levenshtein
distance match. The terms that match help identify the corresponding domain
ontology entry (with its subject-predicate-object triple) is added to the candidate
set. This set of candidate genes is a shortlist of the ’genes’ of the domain that is
probably referred to in the input sentence.
Next, our mechanism evaluates the ‘fittest’ domain ontology entry from the
candidate set to replace the partial gene in the sentence. A fitness function is
defined and evaluated for all the candidate genes short-listed. This is done for
all words / phrases that appear in the input sentence except the noise words.
The fittest genes replace the injured genes of the input sentence. The set of all
genes in the sentence forms the genotypes of the sentence. This is the first-stage of
adaptation.
Once the genotypes are identified, we grow them into phenotypes to remove
the grammatical and linguistic errors in the sentence. To do this, we find parts
of the sentence that is output by the first-stage of adaptation (the gene-level
repair) and that violate well-known grammatical/ linguistic rules. The parts that
violate are repaired through linguistic rules. This is the second stage of adaptation/
repair. This process of artificial rejuvenation improves the accuracy of the sentence,
which can then be processed by a natural language question answering system
(Bhat et al., 2007). Thus, this bio-inspired novel procedure helps adapt/repair
the erroneously recognized text output by a speech recognition engine, in order to
make the output text suitable for deeper natural language processing. The detailed
steps are described below.
Step 1: Genes Identification: We match the sub-parts (or sub-strings) of the

ASR-output sentence with the genes of the domain. The match may be partial
due to the error present in the sentence. The genes in the domain that match
the closest, evaluated by a phonetic and/or syntactic match between the ontology

Adapting general-purpose speech recognition engine for QA

7

The fitness function F takes as input the asr word, the candidate gene, the Levenshtein
distance weight (L), the Phonetic algorithm weight (wP ) and Threshold (T ). Fitness function
F then tries to find the closeness of the match between asr word and the candidate gene. To
do that, the function calculates two scores:
1. algoScore: is an aggregated score of the similarity of the gene with the asr word by various
phonetic algorithms; and
2. editScore: is the Levenshtein distance between the asr word and the gene.
The fitness function then calculates the final fitness of the gene using the formula:
f inalScore = wP ∗ algoScore + L ∗ (1 − editScore).

(1)

If the f inalScore is greater than a given threshold T the asr word is replaced by the candidate
gene, otherwise the asr word is kept as it is, namely,
if (f inalScore > T ) asr word ← gene

Fig. 1 Fitness Function.

entity and the selected sub-part, are picked up and form the candidates set for the
input sentence. For example, let the actual sentence that is spoken by an end-user
be "which industry has the peak sales in nineteen ninety seven?". In one of
our experiments, when Google Speech API was used as the ASR engine for the
above sentence spoken by a user, then the speech engine’s output sentence was
"which industry has the pixel in nineteen ninety seven?". This ASR output
is erroneous (probably due to background noise or the accent of the speaker) and
needs repair/ adaptation for the domain.
As a first step, the ASR output sentence is parsed and the Nouns and Verbs
are identified from part-of-speech (POS) tags. Syntactic parsing also helps get
<subject-verb-object> relations to help identify a potential set of <s-p-o> genes
from the ontology. For each of the Nouns and Verbs and other syntactic relations,
the partially matching genes with respect to the domain ontology are identified; for
this particular sentence the partially matching genes are, "industry" and "pixel".
This leads us to identify the probable set of genes in the domain ontology that
are most likely a possible match: 'INDUSTRY', 'has', 'PEAK SALES'. The set of all
such probable genes need to be evaluated and developed further.
Step 2: Developing the genes to identify the genotypes: Once the basic candidate genes are identified, we evaluate the genes to find the best fit for the situation
on hand with evolution and development of the genes, and then test a fitness function (see Fig. 3.2 and select the most probable gene that survives. This gives us
the set of genotypes that will form the correct ASR sentence. For example, the basic genes "INDUSTRY" and "PIXEL" are used to match the substring "industry has
the pixel" with the gene "INDUSTRY', 'has field', 'PEAK SALES’. This is done
through a matching and fitness function that would identify the most appropriate
gene of the domain. We use a phonetic match function like Soundex, Metaphone,
Double-metaphone (Naumann, 2015) to match "pixel" with "PEAK SALES" or an
edit-distance match function like Levenshtein distance (Naumann, 2015) to find
the closeness of the match. In a large domain there may be many such probable
candidates. In such a case, a fitness function is used to decide which of the matches
are most suitable. The genes identified are now collated together to repair the input

8

Anantaram, Sunil

sentence. This is done by replacing parts of the input sentence by the genes identified in the previous step. In the above example the ASR sentence, "Which industry
has the pixel in nineteen ninety seven?" would be adapted/repaired to "Which
industry has the peak sales in nineteen ninety seven?".
Step 3: Developing Genotypes to Phenotype of sentence: The repaired sen-

tence may need further linguistic adaptation/ repair to remove the remaining
errors in the sentence. To achieve this, the repaired ASR sentence is re-parsed and
the POS tags are evaluated to find any linguistic inconsistencies, and the inconsistencies are then removed. For example, we may notice that there is a WP tag
in a sentence that refers to a Wh-Pronoun, but a WDT tag is missing in the sentence that should provide the Determiner for the Wh-pronoun. Using such clues
we can look for phonetically matching words in the sentence that could possibly
match with a Determiner and repair the sentence. Linguistic repairs such as these
form the genotype to phenotype repair/ adaptation of the sentence. The repaired
sentence can then be processed for question-answering.
We use open source tools like LanguageTool to correct grammatical errors. In
addition we have added some domain specific grammar rules. As we understand,
the LanguageTool has 54 grammar rules, 5 style rules and 4 built-in Python rules
for grammar check and correction. Further we have added some 10 domain specific
rules to our linguistic repair function. Our grammar rules can be extended or
modified for any domain.
3.2.2 Algorithm of the Evo-Devo process

The algorithm has two main functions: ONTOLOGY BASED REPAIR (that encode Steps 1 and 2 described above) and LINGUISTIC REPAIR (encoding Step 3
above). The input sentence is POS tagged and the nouns and verbs are considered.
A sliding window allows the algorithm to consider single words or multiple words
in a domain term.
Let S = w1 , w2 , w3 , · · · , wn be the set of words in the ASR-output(asr out).
Let D = dt1 , dt2 , dt3 , · · · , dtm be the domain-ontology-terms. These terms may be
considered as candidate genes that can possibly replace the ASR output (asr out)
words that may be erroneously recognized. A sliding window of length l consisting
of words wi , · · · , wi+l−1 is considered for matching with domain-ontology-terms.
The length l may vary from 1 to p, where p may be decided based on the environmental information. For example, if the domain under consideration has financial
terms then p may be five words, while for a domain pertaining to car parts, p may
be two words. The part match functionality described below evaluates a cost function, say C ({wi , · · · , wi+l−1 }, dtk ) such that minimizing C ({wi , · · · , wi+l−1 }, dtk )
would result in dt∗ which may be a possible candidate to replace {wi , · · · , wi+l−1 },
namely,
dt∗ = min C ({wi , · · · , wi+l−1 }, dtk )
G

The cost function
C ({wi , · · · , wi+l−1 }, dtk ) =
b1 ∗ soundex(φ{wi , · · · , wi+l−1 }, φ{dtk }) +
b2 ∗ metaphone(φ{wi , · · · , wi+l−1 }, φ{dtk }) +

Adapting general-purpose speech recognition engine for QA

9

b3 ∗ (edit distance(φ{wi , · · · , wi+l−1 }, φ{dtk }) +
b4 ∗ (number of syllables(φ{wi , · · · , wi+l−1 }) − number of syllables(φ{dtk }) +
b5 ∗ (word2vec(φ{wi , · · · , wi+l−1 }) − word2vec(φ{dtk })2

where weights, b1 + b2 + b3 + b4 + b5 = 1 and φ{k} represents each-element-in
the set k. If the value of the cost function C ({wi , · · · , wi+l−1 }, dtk ) is greater than
the pre-determined threshold T then {wi , · · · , wi+l−1 } may be replaced with the
dt∗ , otherwise the {wi , · · · , wi+l−1 } is maintained as it is. The broad algorithm of
Evolutionary Development mechanism is shown in Algorithm 1.

Algorithm 1 Evo-Devo Mechanism
INPUT: ASR output sentence, sentence; domain ontology
OUTPUT: Repaired sentence, repaired sentence
start
1: // parse the input sentence
2: parsed sentence ← POS tag(sentence)
3: // repair process starts - do genetic repair and find the genotypes
4: part repaired sentence ← ontology based repair(parsed sentence)
5: // grow the genotypes into phenotypes
6: repaired sentence ← linguistic repair(parsed sentence,part repaired sentence)
end
7: function ontology based repair(parsed sentence)
8:
nouns verbs ← find(parsed sentence, noun verb POStoken)
9: // for each noun verb entry in nouns verbs do next 4 steps
10: // find partially matching genes: match nouns and verbs with entries in domain ontology
with phonetic algorithms and Levenshtein distance match
11:
concepts referred ← part match(noun verb entry, domain ontology)
12: // find genes: get the subject-predicate-object for concepts
13:
candidate genes ← add(spo entry, concepts referred)
14: // simulate the development process of the genes - find the fittest gene from candidate
genes
15:
fit gene ← fittest(candidate genes, POS token)
16: // add fittest gene into set of genotypes
17:
genotypes ← add(fit gene)
18: // replace partially identified genes in input with genotypes identified
19:
repaired sentence ← substitute(parsed sentence, nouns verbs, genotypes)
20:
return repaired sentence
21: end function
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:

function linguistic repair(part repaired sentence)
other POStags ← find(part repaired sentence, remaining POStokens)
// find POS tags without linguistic completion
ling err ← linguistic check( other POStags, part repaired sentence)
// find candidate words for linguistic error
candidate words ← add(part repaired sentence, ling err)
// find the closest semantic match for error words
fit word ← fittest word(candidate words, ling err)
// add fittest word into repaired sentence
fit words ← add(candidate word, fit word)
// create the repaired sentence
repaired sentence ← replace(part repaired sentence, fit words, other POStags)
return repaired sentence
end function

10

Anantaram, Sunil

3.2.3 Detailed example of our method

Let us assume that we have the domain of retail sales data described in an ontology
of <subject-predicate-object> structure as shown in Table 1.
Subject
INDUSTRY
INDUSTRY
CAR DEALERS
CAR DEALERS
CAR DEALERS
OPTICAL GOODS
OPTICAL GOODS
OPTICAL GOODS

Predicate
BUSINESS
BUSINESS
SALES 2013
SALES 2011
SALES 2009
SALES 2013
SALES 2011
SALES 2009

Object
CAR DEALERS
OPTICAL GOODS
737640 million
610747 million
486896 million
10364 million
10056 million
8767 million

Table 1 Ontology Structure.

Now, let us consider that a user speaks the following sentence to Google
Now speech engine: "Which business has more sales in 2013: Car dealers or
optical goods?". In our experiment the Google Now speech engine produced the
ASR output sentence as "which business has more sales in 2013 car dealers
for optical quotes". The recognized ASR sentence has errors. In order to make
this ASR sentence more accurate, we input this sentence into the Evo-Devo mechanism and run the process:
– Genes Identification (Step 1): We parse the ASR sentence and identify the
parts-of-speech in it as: which/WDT, business/NN, has/VBZ, more/JJR, sales/NNS,
in/IN, 2013/CD, car/NN, dealers/NNS, for/IN, optical/JJ, quotes/NNS.
Considering the words that have POS tags of Nouns (NN/NNS etc.) in the
example sentence we get the words "business", "sales", "car", "dealers",
"quotes". Based on these words we extract all the partially matching subject-

predicate-object instances of the domain ontology. For example, we obtain instances such as [OPTICAL GOODS SALES 2013 10364 million], [INDUSTRY BUSINESS
OPTICAL GOODS] and [INDUSTRY BUSINESS CAR DEALERS], etc. from the domain
ontology that are partially matching with the words "business" and "sales"
respectively. POS tag 2013/CD also leads to reinforcing the above <s-p-o> instance.
– Developing the genes to identify the genotypes (Step 2): We replace
the erroneous words in the sentence by using a fitness function. The fitness
function is defined using string similarity metric (Levenshtein distance) and
an aggregated score of phonetic algorithms such as Soundex, Metaphone and
Double Metaphone as described in Fitness function in the section above. Thus
we get the following adaptation: which business has more sales in 2013 car
dealers for optical goods?
– Developing Genotypes to Phenotype (Step 3): We now find the parts-ofspeech of the repaired sentence after the step 2 as: which/WDT, business/NN,
has/VBZ, more/JJR, sales/NNS, in/IN, 2013/CD, car/NN, dealers/NNS, for/IN,
optical/JJ, goods/NNS.

In the linguistic repair step, we find that since there is no direct ontological
relationship between "car dealers" and "optical goods", we cannot have the

Adapting general-purpose speech recognition engine for QA

11

preposition for between these domain terms. Thus we have to find a linguistic
relation that is permissible between these domain terms. One of the options
is to consider linguistic relations like ‘conjunction’, ‘disjunction’ between domain terms. Thus, when we evaluate linguistic relations AND or OR between
these domain terms, we find that OR matches closely with for through a phonetic match rather than AND. Thus we replace for with or in the sentence.
Hence the final output of the Evo-Devo mechanism is "which business has
more sales in 2013 car dealers or optical goods?". This sentence can now
be processed by a question-answering (QA) system. In the above example, a QA
system (Bhat et al., 2007) would parse the sentence, identify the known ontological terms {business, sales, 2013, car dealers, optical goods}, find
the unknown predicates {which business, more sales}, form the appropriate
query over the ontology, and return the answer "CAR DEALERS".
3.2.4 Limitations of the method

We assume that there is a well-structured domain ontology for the domain and
it is available in the form of <s-p-o> triples. We also assume that the speaker
speaks mostly grammatically correct sentences using terms in the domain. While
the method would work for grammatically incorrect sentences, the linguistic repair
step would suffer.
We assume that the speech is processed by a gp-ASR and the ASR-output
forms the input sentence that needs repair. However, it is important to note that
the input sentence (i.e. the ASR output) need not necessarily contain <s-p-o>
triples for our method to work. The <s-p-o> triples that are short-listed from domain ontology aid in forming a candidate set of ’possible genes’ to consider and the
fittest amongst them is considered (Step 2) in the context of the other words in the
sentence. For example, if the input sentence was ’Who had pick sales’ would get
repaired to ’Who had peak sales’ since the domain term of ’peak sales’ would
match with ’pick sales’ in our method. Further, input sentences need not necessarily be queries; these can be just statements about a domain. For example, if the
above ASR-output sentence was "We hit pick sales this season", the method
would repair it as "We hit peak sales this season" using the same set of steps
for repair. However, as of now, our method does not repair paraphrases of sentences
like "which industry had biggest sales" to "which industry had peak sales".
Such repairs need extension to our matching process.
The method does not impose any restriction on the sentence or its formation; it
can be a fully meaningful sentence in a domain or may contain partial information.
The method finds the fittest repair for the inaccuracies occurring in an sentence,
post-ASR recognition. It should also be noted that the method does not know the
original sentence spoken by the speaker, but tries to get back the original sentence
for a particular domain.

3.3 Machine Learning mechanism of adaptation
In the machine learning based mechanism of adaptation, we assume the availability
of example pairs of (T 0 , T ) namely (ASR output, the actual transcription of the
spoken sentence) for training. We further assume that such a machine-learnt model

12

Anantaram, Sunil

can help repair an unseen ASR output to its intended correct sentence. We address
the following hypothesis
Using the information from past recorded errors and the corresponding correction, can we learn how to repair (and thus adapt to a new domain) the text
after ASR?

Note that this is equivalent to, albiet loosely, learning the error model of a specific
ASR. Since we have a small training set, we have used the Naive Bayes classifier
that is known to perform well for small datasets with high bias and low variance. We have used the NLTK (Bird et al., 2009) Naive Bayes classifier in all our
experiments.
Let T 0 be the erroneous text (which is the ASR output), T the corresponding
reference text (which is the textual representation of the spoken sentence) and F
a feature extractor, such that
fβ = F (T 0 β )

(2)

fβ = (fβ 1 , fβ 2 , · · · fβn )

(3)

where
0

is a set of n features extracted from T β . Suppose there are several pairs say (T 0 i ,
Ti ) for i = 1, 2, · · ·N . Then we can derive fi for each T 0 i using (2). The probability
that T 0 k belongs to the class Tk can be derived through the feature set fk as
follows.
P (Tk ) ∗ P (fk |Tk )
P (Tk |fk ) =
P (f k )
where P (Tk ) is the apriori probability of the class Tk and P (fk |Tk ) is the probability
of occurrence of the features fk in the class Tk , and P (fk ) is the overall probability
of the occurrence of the feature set fk . Making naive assumption of independence
in the features fk1 , fk2 , · · ·fkn we get
P (Tk |fk ) =

P (Tk ) ∗ (P (fk1 |Tk ) ∗ P (fk2 |Tk ) ∗ · · ·P (fkn |Tk ))
P (f k )

(4)

In our experiments, the domain specific reference text Ti was spoken by several
people and the spoken speech was passed through a general purpose speech recognition engine (ASR) that produced a (possibly) erroneous hypothesis T 0 i . Each
pair of reference and the ASR output (i.e. hypothesis) was then word aligned using edit distance, and the mismatching pairs of words were extracted as (T 0 i , Ti )
pairs. For example, if we have the following spoken sentence:

/In which year beer wine and liquor stores has
S1 :
successful year/

and the corresponding true transcription

In which year beer wine and liquor stores has
T1 :
successful year

One of the corresponding ASR output T 0 1 was

in which year dear wine and liquor stores have
T 01 :
successful year

Adapting general-purpose speech recognition engine for QA

13

In this case the (T 0 , T ) pairs are (dear, beer) and (have, has). As another
example consider that T2 was spoken but T 0 2 was recognized by the ASR.

T2 :

T 02 :

Whether the sales of jewelry business crosses fifty
thousand in a year



whether the sales of than twenty business crosses fifty
thousand in a year

Clearly, in this case the (T 0 , T ) pair is (than twenty, jewelry).
Let us assume two features, namely, fβ in (2) is of dimension n = 2. Let the
two features be (number of words, number of syllables). Then, for the (T 0 , T ) pair
(than twenty, jewelry) we have
F ((than twenty)) = (2, 3)

since the number of words in than twenty is 2 and than twenty contains 3 syllables.
P (fk1 |Tk ) in this case would be the probability that the number of words in the
input are two (fk1 = 2) when the correction is jewelry. A third example is:
T3 :

n

In two thousand thirteen which industry had the peak
sales

T 0 3 : {in two thousand thirteen which industry have the pixels

Note that in this case the (T 0 , T ) pair is (peak sales, pixel).
Calculating thus the values of P (Tk ) for all reference corrections, P (fkj |Tk ) for
all feature values, fkj for all the j features in fk , we are in a position to calculate
the RHS of (4). When this trained classifier is given an erroneous text, features
are extracted from this text and the repair works by replacing the erroneous word
by a correction that maximizes (4),
Tk∗ = max P (Tk |fk )
Tk

Namely, the Tk∗ for which P (Tk |fk ) is maximum.

4 Experiments and results

We present the results of our experiments with both the Evo-Devo and the Machine
Learning mechanisms described earlier using the U.S. Census Bureau conducted
Annual Retail Trade Survey of U.S. Retail and Food Services Firms for the period
of 1992 to 2013 (USCensus, 2015).

14

Anantaram, Sunil

Fig. 2 T 0 accuracy (y-axis) for the 250 utterance (x-axis) for Ga, Ki, Ku and Ps.

4.1 Data Preparation
We downloaded this survey data and hand crafted a total of 293 textual questions
(AwazYP, 2015) which could answer the survey data. A set of 6 people (L2 English)
generated 50 queries each with the only constraint that these queries should be
able to answer the survey data. In all a set of 300 queries were crafted of which
duplicate queries were removed to leave 293 queries in all. Of these, we chose 250
queries randomly and distributed among 5 Indian speakers, who were asked to
read aloud the queries into a custom-built audio data collecting application. So,
in all we had access to 250 audio queries spoken by 5 different Indian speakers;
each speaking 50 queries.
Each of these 250 audio utterances were passed through 4 different ASR engines, namely, Google ASR (Ga), Kaldi with US acoustic models (Ku), Kaldi with
Indian Acoustic models (Ki) and PocketSphinx ASR (Ps). In particular, that
audio utterances were in wave format (.wav) with a sampling rate of 8 kHz and
16 bit. In case of Google ASR (Ga), each utterance was first converted into .flac
format using the utility sound exchange (sox) commonly available on Unix machines. The .flac audio files were sent to the cloud based Google ASR (Ga) one
by one in a batch mode and the text string returned by Ga was stored. In all 7
utterances did not get any text output, presumably Ga was unable to recognize
the utterance. For all the other 243 utterances a text output was received.
In case of the other ASR engines, namely, Kaldi with US acoustic models (Ku),
Kaldi with Indian Acoustic models (Ki) and PocketSphinx ASR (Ps) we first took
the queries corresponding to the 250 utterances and built a statistical language
model (SLM) and a lexicon using the scripts that are available with PocketSphinx

Adapting general-purpose speech recognition engine for QA

15

S : {/Which stores has total sales more than two hundred thousand/
T 0 : {which state has total sales more than twenty thousand

Ga; 70.00%

0

T : {which stores has total sales more than two in two thousand

Ki; 80.00%

T 0 : {which stores has total sales more than point of sales

Ku; 70.00%

T 0 : {list the total sales more than

Ps; 40.00%

Fig. 3 Sample output (T 0 ) of four different ASR for the same spoken utterance (S). Also
shown are the accuracy of the ASR output.
ASR engine

Result

No result

Google ASR (Ga)
Kaldi US (Ku)
Kaldi IN (Ki)
PocketSphinx (Ps)

243
250
250
250

7
0
0
0

Total

993

7

Correct
55
103
127
44

Error
(A+B)
188
147
123
206

>=70%
(A)
143
123
111
109

<70%
(B)
45
24
12
97

329

664

486

178

Table 2 ASR engines and their output %accuracy

(CMU, 2017) and Kaldi (Kaldi, 2017). This language model and lexicon was used
with the acoustic model that were readily available with Kaldi and Ps. In case of
Ku we used the American English acoustic models, while in case of Ki we used
the Indian English acoustic model. In case of Ps we used the Voxforge acoustic
models (VoxForge, 2017). Each utterance was passed through Kaldi ASR for two
different acoustic models to get T 0 corresponding to Ku and Ki. Similarly all the
250 audio utterance were passed through the Ps ASR to get the corresponding T 0
for Ps. A sample utterance and the output of the four engines is shown in Figure
3.
Figure 2 and Table 2 capture the performance of the different speech recognition engines. The performance of the ASR engines varied, with Ki performing the
best with 127 of the 250 utterances being correctly recognized while Ps returned
only 44 correctly recognized utterances (see Table 2, Column 4 named ”Correct”)
of 250 utterances. The accuracy of the ASR varied widely. For instance, in case of
Ps there were as many as 97 instances of the 206 erroneously recognized utterances
which had an accuracy of less than 70%.
Note that the accuracy is computed as the number of deletions, insertions,
substitutions that are required to convert the ASR output to the textual reference
(namely, T 0 → T ) and is a common metric used in speech literature (Hunt, 1990).
For all our analysis, we used only those utterances that had an accuracy 70%
but less that 100%, namely, 486 instances (see Table 2, Figure 4). An example
showing the same utterance being recognized by four different ASR engines is
shown in Figure 3. Note that we used T 0 corresponding to Ga, Ki and Ku in our
analysis (accuracy ≥ 70%) and not T 0 corresponding to Ps which has an accuracy
of 40% only. This is based on our observation that any ASR output that is lower
that 70% accurate is so erroneous that it is not possible to adapt and steer it
towards the expected output. The ASR output (T 0 ) are then given as input in
the Evo-Devo and Machine Learning mechanism of adaptation.

16

Anantaram, Sunil

Fig. 4 All utterances that have and T 0 accuracy (y-axis) ≥ 70 and < 100 used in all our
experiments.

4.2 Evo-Devo based experiments
We ran our Evo-Devo mechanism with the 486 ASR sentences (see Table 2) and
measured the accuracy after each repair. On an average we have achieved about
5 to 10% improvements in the accuracy of the sentences. Fine-tuning the repair
and fitness functions, namely Equation (1), would probably yield much better
performance accuracies. However, experimental results confirm that the proposed
Evo-Devo mechanism is an approach that is able to adapt T 0 to get closer to T .
We present a snapshot of the experiments with Google ASR (Ga) and calculate
accuracy with respect to the user spoken question as shown in Table 3.
Table 3 clearly demonstrates the promise of the evo-devo mechanism for adaptation/repair. In our experiments we observed that the adaptation/repair of subparts in ASR-output (T 0 ) that most probably referred to domain terms occurred
well and were easily repaired, thus contributing to increase in accuracy. For nondomain-specific linguistic terms the method requires one to build very good linguistic repair rules, without which the method could lead to a decrease in accuracy.
One may need to fine-tune the repair, match and fitness functions for linguistic
terms. However, we find the abstraction of evo-devo mechanism is very apt to use.

4.3 Machine Learning experiments
In the machine learning technique of adaptation, we considers (T 0 , T ) pairs as the
predominant entity and tests the accuracy of classification of errors.

Adapting general-purpose speech recognition engine for QA
User’s Question (T ), Google ASR out (T 0 Ga ), After Evo-devo (T 0 ED )
T : {In two thousand fourteen which industry had the peak sales
T 0 Ga : {in two thousand fourteen which industry had the pixels
T 0 ED : {in two thousand fourteen which industry had the peak sales
nin which year did direct selling establishments make the
T :
maximum
and in which year did they do the minimum sales
nwhichsales
year did direct selling establishments make the maximum
T 0 Ga :
and in which year did they do the many muscles
ncells
which year did direct selling establishments make the maximum
T 0 ED :
sales and in which year did they do the many musical
(Which one among the electronics and appliance store and food
T : and beverage stores has sales in more than hundred thousand
in(at least three years in a row
which one among the electronics and appliance store and food
T 0 Ga : and beverages stores have sales in more than one lakh in at
least three years in a row
(which
one among the electronics and appliance store and food
0
T ED : and beverage stores have sales in more than one lakh in at
least three years in a row

17
Acc
Ga: 80%
ED: 100%

Ga:80.9%
ED: 85.7%

Ga: 85.7%
ED:89.3%

Table 3 Evo-Devo experiments with Google ASR (Ga).

In our experiment, we used a total of 570 misrecognition errors (for example,
(dear, beer) and (have, has) derived from (T 0 1 , T1 ) or (than twenty, jewelry)
derived from (T 0 2 , T2 )) in the 486 sentences. We performed 10-fold cross validation,
each fold containing 513 (T 0 , T ) pairs for training and 57 pairs for testing, Note

that we assume the erroneous words in the ASR output being marked by a human
oracle, in the training as well as the testing set. Suppose the following example
(T4 ) occurs in the training set:
T4 :

n

T 04 :

n

Which business has posted cumulative sales of more than
one million dollars from the 2007 to 2012
which business has posted latest stills of more than
one million dollars from 2007 to 2012

The classifier is given the pair {F (latest stills), cumulative sales} to the classifier. And if the following example occurs in the testing set (T5 ),
T5 :

n

T 05 :

n

Did sales remain the same in retail between two
thousand thirteen and two thousand fourteen
sales wine same in retail between two thousand thirteen
and two thousand fourteen

the trained model or the classifier is provided F (wine) and successful repair would
mean it correctly labels (adapts) it to remain the. The features used for classification were (n = 6 in Equation (3))
fβ 1
fβ 2
fβ 3
fβ 4
fβ 5
fβ 6

→
→
→
→
→
→

Left context (word to the left of T 0 ),
Number of errors in the entire ASR sentence,
Number of words in T 0 ,
Right context (word to the right of T 0 ),
Bag of vowels of T 0 and
Bag of consonants of T 0 .

18

Anantaram, Sunil

The combination of features fβ 6 , fβ 5 , fβ 1 , fβ 3 , fβ 4 namely, (bag of consonants,
bag of vowels, left context, number of words, right context) gave the best results
with 32.28% improvement in accuracy in classification over 10-fold validation.
The experimental results for both evo-devo and machine learning based approaches demonstrate that these techniques can be used to correct the erroneous
output of ASR. This is what we set out to establish in this paper.

5 Conclusions

General-purpose ASR engines when used for enterprise domains may output erroneous text, especially when encountering domain-specific terms. One may have
to adapt/repair the ASR output in order to do further natural language processing such as question-answering. We have presented two mechanisms for adaptation/repair of ASR-output with respect to a domain. The Evo-Devo mechanism
provides a bio-inspired abstraction to help structure the adaptation and repair
process. This is one of the main contribution of this paper. The machine learning
mechanism provides a means of adaptation and repair by examining the featurespace of the ASR output. The results of the experiments show that both these
mechanisms are promising and may need further development.

6 Acknowledgments

Nikhil, Chirag, Aditya have contributed in conducting some of the experiments.
We acknowledge their contribution.

References

Cheongjae Lee, Sangkeun Jung, Kyungduk Kim, Donghyeon Lee, and Gary Geunbae Lee. Recent approaches to dialog management for spoken dialog systems.
Journal of Computing Science and Engineering, 4(1):1–22, 2010.
C. Anantaram, Rishabh Gupta, Nikhil Kini, and Sunil Kumar Kopparapu. Adapting general-purpose speech recognition engine output for domain-specific natural
language question answering. In Workshop on Replicability and Reproducibility in
Natural Language Processing: adaptive methods, resources and software at IJCAI
2015, Buenos Aires, 2015a.

E.K. Ringger and J.F. Allen. Error correction via a post-processor for continuous
speech recognition. In Acoustics, Speech, and Signal Processing, 1996. ICASSP-96.
Conference Proceedings., 1996 IEEE International Conference on, volume 1, pages
427–430 vol. 1, May 1996. doi: 10.1109/ICASSP.1996.541124.
Yongmei Shi. An Investigation of Linguistic Information for Speech Recognition Error
Detection. PhD thesis, University of Maryland, Baltimore County, October 2008.
Lina Zhou, Jinjuan Feng, A. Sears, and Yongmei Shi. Applying the naive bayes
classifier to assist users in detecting speech recognition errors. In System Sciences,
2005. HICSS ’05. Proceedings of the 38th Annual Hawaii International Conference
on, pages 183b–183b, Jan 2005. doi: 10.1109/HICSS.2005.99.

Adapting general-purpose speech recognition engine for QA

19

Minwoo Jeong, Byeongchang Kim, and G Lee. Using higher-level linguistic knowledge for speech recognition error correction in a spoken q/a dialog. In HLTNAACL special workshop on Higher-Level Linguistic Information for Speech Processing, pages 48–55, 2004.

Ramón López-Cózar and Zoraida Callejas. Asr post-correction for spoken dialogue systems based on semantic, syntactic, lexical and contextual information.
Speech Commun., 50(8-9):745–766, August 2008. ISSN 0167-6393. doi: 10.1016/j.
specom.2008.03.008. URL http://dx.doi.org/10.1016/j.specom.2008.03.008.
Youssef Bassil and Paul Semaan. ASR context-sensitive error correction based on
microsoft n-gram dataset. CoRR, abs/1203.5262, 2012. URL http://arxiv.org/
abs/1203.5262.
J. Jun and L. Lei. Asr post-processing correction based on ner and pronunciation
primitive. In 2011 7th International Conference on Natural Language Processing
and Knowledge Engineering, pages 126–131, Nov 2011. doi: 10.1109/NLPKE.
2011.6138180.
B. Amento, P. Isenhour, and L. Stead. Error correction in automatic speech recognition transcripts, September 6 2007. URL https://www.google.com/patents/
US20070208567. US Patent App. 11/276,476.
David Harwath, Alexander Gruenstein, and Ian McGraw. ”choosing useful
word alternates for automatic speech recognition correction interfaces. In
INTERSPEECH-2014, pages 949–953, 2014.
Simon Harding and Wolfgang Banzhaf. Artificial development. In Organic Computing, Understanding Complex Systems, pages 201–219. Springer Berlin Heidelberg, 2008. ISBN 978-3-540-77656-7. doi: 10.1007/978-3-540-77657-4 9. URL
http://dx.doi.org/10.1007/978-3-540-77657-4_9.
C Anantaram, Nikhil Kini, Chirag Patel, and Sunil Kopparapu. Improving asr
recognized speech output for effective nlp. In The Ninth International Conference
on Digital Society, pages 17–21, Lisbon, Portugal, Feb 2015b.
Gunnar Tufte. From Evo to EvoDevo: Mapping and Adaptation in Artificial
Development. Development, 2008.
Shefali Bhat, C. Anantaram, and Hemant Jain. Framework for text-based conversational user-interface for business applications. In Proceedings of the 2Nd
International Conference on Knowledge Science, Engineering and Management,
KSEM’07, pages 301–312, Berlin, Heidelberg, 2007. Springer-Verlag. ISBN
3-540-76718-5, 978-3-540-76718-3. URL http://dl.acm.org/citation.cfm?id=
1775431.1775465.
Felix Naumann. http://hpi.de/fileadmin/user_upload/fachgebiete/naumann/
folien/SS13/DPDC/DPDC_12_Similarity.pdf, 2015.
Steven Bird, Ewan Klein, and Edward Loper. Natural Language Processing
with Python. O’Reilly Media, Inc., 1st edition, 2009. ISBN 0596516495,
9780596516499.
USCensus. http://www.census.gov/retail/index.html, 2015. Viewed Sep 2015.
AwazYP. https://sites.google.com/site/awazyp/data/ijcai, 2015. Viewed Aug
2017.
CMU. Building language model for pocketsphinx, 2017. URL http://cmusphinx.
sourceforge.net/wiki/tutoriallm.
Kaldi. Overview of graph creation in kaldi, 2017. URL http://kaldi-asr.org/
doc/graph.html.

20

Anantaram, Sunil

VoxForge. Updated 8khz sphinx acoustic model, 2017. URL http://www.voxforge.
org/home/news/news/updated-8khz-sphinx-acoustic-model.
Melvyn J. Hunt. Figures of merit for assessing connected-word recognisers. Speech
Communication, 9(4):329 – 336, 1990. ISSN 0167-6393. doi: http://dx.doi.org/10.
1016/0167-6393(90)90008-W. URL http://www.sciencedirect.com/science/
article/pii/016763939090008W.

