FMLLR

In signal processing, Feature space Maximum Likelihood Linear Regression (fMLLR) is a global feature transform that are typically applied in a speaker adaptive way, where fMLLR transforms acoustic features to speaker adapted features by a multiplication operation with a transformation matrix. In some literature, fMLLR is also known as the Constrained Maximum Likelihood Linear Regression (cMLLR).
fMLLR transformations are trained in a maximum likelihood sense on adaptation data. These transformations may be estimated in many ways, but only maximum likelihood (ML) estimation is considered in fMLLR. The fMLLR transformation is trained on a particular set of adaptation data, such that it maximizes the likelihood of that adaptation data given a current model-set.
This technique is a widely used approach for speaker adaptation in HMM-based speech recognition,
later research also shows that fMLLR is an excellent acoustic feature for DNN/HMM hybrid speech recognition models.
The advantage of fMLLR includes the following:
Major problem and disadvantage of fMLLR:
Computing fMLLR transform.
Feature transform of fMLLR can be easily computed with the open source speech tool Kaldi, the Kaldi script uses the standard estimation scheme described in Appendix B of the original paper, in particular the section Appendix B.1 "Direct method over rows".
In the Kaldi formulation, fMLLR is an affine feature transform of the form formula_1→formula_2formula_1formula_4, which can be written in the form formula_1→Wformula_6, where formula_6 =formula_8 is the acoustic feature formula_1 with a 1 appended. Note that this differs from some of the literature where the 1 comes first as formula_6 =formula_11.
The sufficient statistics stored are:
formula_12
where formula_13 is the inverse co-variance matrix.
And for formula_14 where formula_15 is the feature dimension:
formula_16
For a thorough review that explains fMLLR and the commonly used estimation techniques, see the original paper "Maximum likelihood linear transformations for HMM-based speech recognition ".
Note that the Kaldi script that performs the feature transforms of fMLLR differs with by using a column of the inverse in place of the cofactor row. In other words, the factor of the determinant is ignored, as it does not affect the transform result and can causes potential danger of numerical underflow or overflow.
Comparing with other features or transforms.
Experiment result shows that by using the fMLLR feature in speech recognition, constant improvement is gained over other acoustic features on various commonly used benchmark datasets (TIMIT, LibriSpeech, etc).
In particular, fMLLR features outperform MFCCs and FBANKs coefficients, which is mainly due to the speaker adaptation process that fMLLR performs.
In, phoneme error rate (PER, %) is reported for the test set of TIMIT with various neural architectures:
As expected, fMLLR features outperform MFCCs and FBANKs coefficients despite the use of different model architecture.
Where MLP (multi-layer perceptron) serves as a simple baseline, on the other hand RNN, LSTM, and GRU are all well known recurrent models.
The Li-GRU architecture is based on a single gate and thus saves 33% of the computations over a standard GRU model, Li-GRU thus effectively address the gradient vanishing problem of recurrent models.
As a result, the best performance is obtained with the Li-GRU model on fMLLR features.
Extract fMLLR features with Kaldi.
fMLLR can be extracted as reported in the s5 recipe of Kaldi.
Kaldi scripts can certainly extract fMLLR features on different dataset, below are the basic example steps to extract fMLLR features from the open source speech corpora Librispeech.
Note that the instructions below are for the subsets codice_1,codice_2,codice_3, and codice_4,
but they can be easily extended to support the other sets codice_5, codice_6, and codice_7
export train_cmd="run.pl --mem 2G"
export decode_cmd="run.pl --mem 4G"
export mkgraph_cmd="run.pl --mem 8G"
data=/media/user/SSD # example path
mkdir exp/tri4b/decode_tgsmall_train_clean_100 && cp exp/tri4b/trans.* exp/tri4b/decode_tgsmall_train_clean_100/
. ./cmd.sh ## You'll want to change cmd.sh to something that will work on your system.
. ./path.sh ## Source the tools/utils (import the queue.pl)
gmmdir=exp/tri4b
for chunk in dev_clean test_clean train_clean_100 train_clean_360 ; do
 dir=fmllr/$chunk
 steps/nnet/make_fmllr_feats.sh --nj 10 --cmd "$train_cmd" \
 --transform-dir $gmmdir/decode_tgsmall_$chunk \
 $dir data/$chunk $gmmdir $dir/log $dir/data || exit 1
 compute-cmvn-stats --spk2utt=ark:data/$chunk/spk2utt scp:fmllr/$chunk/feats.scp ark:$dir/data/cmvn_speaker.ark
done
steps/align_fmllr.sh --nj 10 data/dev_clean data/lang exp/tri4b exp/tri4b_ali_dev_clean
steps/align_fmllr.sh --nj 10 data/test_clean data/lang exp/tri4b exp/tri4b_ali_test_clean
steps/align_fmllr.sh --nj 30 data/train_clean_100 data/lang exp/tri4b exp/tri4b_ali_clean_100
steps/align_fmllr.sh --nj 30 data/train_clean_360 data/lang exp/tri4b exp/tri4b_ali_clean_360
data=/user/kaldi/egs/librispeech/s5 ## You'll want to change this path to something that will work on your system.
rm -rf $data/fmllr_cmvn/
mkdir $data/fmllr_cmvn/
for part in dev_clean test_clean train_clean_100 train_clean_360; do
 mkdir $data/fmllr_cmvn/$part/
 apply-cmvn --utt2spk=ark:$data/fmllr/$part/utt2spk ark:$data/fmllr/$part/data/cmvn_speaker.ark scp:$data/fmllr/$part/feats.scp ark:- | add-deltas --delta-order=0 ark:- ark:$data/fmllr_cmvn/$part/fmllr_cmvn.ark
done
du -sh $data/fmllr_cmvn/*
echo "Done!"
python ark2libri.py