User Tools

Site Tools


sphinxtrainwalkthrough

This document contains an overview of SphinxTrain from the perspective of researchers or developers wishing to implement new features or training methods. The process of training is already covered by the Robust Tutorial and the Sphinx Manual, so it will not be covered here except as necessary.

Layout of SphinxTrain code

The SphinxTrain code is organized into a few static libraries which contain most of the “core” functionality, and a large number of tools which do manipulations on acoustic model files. In many cases, the tools mostly just call library functions and don't themselves contain a lot of code.

Python Modules

The Python modules are located in the 'python/sphinx' directory of the source tree. There is a setup.py file in the 'python' top-level directory which can be used to install these modules. However, since all modules are written purely in Python, you can also simply set up your sys.path variable to point to the source directory (provided that you have first installed NumPy):

#! python
dhuggins@slim:~/Projects/Sphinx/SphinxTrain$ python
Python 2.5.1 (r251:54863, Oct  5 2007, 13:36:32)
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.insert(0, "python/sphinx")
>>> from sphinx import *
>>>

Some automatically-generated (and incomplete) documentation on the Python modules can be found at http://lima.lti.cs.cmu.edu/pydoc/SphinxTrain/

Header Files

Nearly all of the header files useful in SphinxTrain development are in the 'include/s3' directory.

Libraries

The current directory layout of the library portion of the SphinxTrain code is:

src/libs/libcommon
src/libs/libio
src/libs/libs2io
src/libs/libcep_feat
src/libs/libmodinv
src/libs/libmllr
src/libs/libclust
src/libs/librpcc

A description of these libraries follows:

* libcommon: contains utility functions (many of which replicate or reimplement similar functions in SphinxBase) * libio: contains reading and writing functions for every sort of file type dealt with in SphinxTrain. * libs2io: contains reading and writing functions (or actually just writing, I think) for Sphinx-II format model files * libcep_feat: contains dynamic feature computation code. This does the same thing (but with a different API) as libsphinxfeat in SphinxBase. * libmodinv: contains model inventory (i.e. acoustic model) functions, including Gaussian density and mixture model computation. * libmllr: contains MLLR adaptation functions (which ought to be in libmodinv, probably) * libclust: contains clustering functions, primarily related to decision tree building for state tying * librpcc: is obsolete (formerly contained one function, which reads the performance counter on DEC Alphas using the RPCC instruction) Most of the code that you are likely to want to modify is in libmodinv and possibly also libclust and libcep_feat.

Tools

Source code for the training programs is in the 'src/programs' directory. The functionality of most of these programs is detailed elsewhere. The ones which are the most important for the training process are 'bw' and 'norm'. The 'bw' program collects expected state occupation and transition counts using the Forward-Backward algorithm, while 'norm' performs Maximum Likelihood updating of acoustic model parameters based on these counts.

Writing SphinxTrain programs in C

Basic structure of a SphinxTrain tool

Let's look at the 'norm' tool to demonstrate the typical structure of a SphinxTrain tool written in C. This tool's source code is in the directory 'src/programs/norm' inside the SphinxTrain source tree, and contains the following files:

src/programs/norm/main.c
src/programs/norm/Makefile
src/programs/norm/parse_cmd_ln.c
src/programs/norm/parse_cmd_ln.h

First the file 'Makefile' contains instructions for compiling this tool. Most of the build logic is contained in the file config/common_make_rules which is included at the end of each tool's Makefile. The Makefile simply defines a number of variables which describe the source files. In the future, we may switch to using GNU Autotools in which case there will be a 'Makefile.am' which operates similarly. Here is the important part of the Makefile:

TOP=../../..
DIRNAME=src/programs/norm
BUILD_DIRS =
ALL_DIRS= $(BUILD_DIRS)
SRCS = \
  main.c \
  parse_cmd_ln.c
H = \
  parse_cmd_ln.h
FILES = Makefile $(SRCS) $(H)
TARGET = norm
ALL = $(BINDIR)/$(TARGET)
include $(TOP)/config/common_make_rules

The variables TOP and DIRNAME are required in all Makefiles in SphinxTrain. BUILD_DIRS and ALL_DIRS are only needed if there are subdirectories within the Makefile's directory which need to be built. The SRCS variable lists all C source files to be compiled while the H variable lists all the header files in this directory. The FILES directory lists all files which should be included in a distribution package. The TARGET variable is very important - it gives the name of the program which is to be built from the source files in this directory. The final two lines are required in order to make everything work.

Now, let's look at parse_cmd_ln.c and parse_cmd_ln.h. The main function of these source files is to defined the command-line arguments to the tool. For whatever historical reasons, these two files also contain a bunch of boilerplate code which is duplicated in every tool, with the only changes being the actual definition of arguments and the help text and description of the tool. Again, this may change in the near future. Finally the main.c file contains the actual code. Let's walk through what this does. First, the initialize() function parses the command-line:

#! cplusplus
static int
initialize(int argc,
           char *argv[])
{
    /* define, parse and (partially) validate the command line */
    parse_cmd_ln(argc, argv);
    return S3_SUCCESS;
}

This sets the internal variables which are read by the various functions in <s3/cmd_ln.h> such as cmd_ln_str(), cmd_ln_float32(), and such. Unfortunately these access macros are not used consistently in SphinxTrain, so you will see a lot of code that just uses cmd_ln_access() and casts the result to some other type. This is BAD and should not be done, because in SphinxBase the return value of cmd_ln_access() is not a simple void pointer.

The actual work of the 'norm' program is done in the normalize() function. There is no good reason to structure your code this way - you could just as easily do all this stuff in the main() function, or preferably, you could split this up into a number of smaller funtcions. But for whatever reason, this is the way that 'norm' was written, so let's look at what normalize() does. First, it declares an utterly ridiculous number of variables. This is not accepted best practice for programming in the 21st century. So, we'll skip that part. Most of these variables are simply used to hold values acquired from the command-line. One important thing to note is that the command-line variable -accumdir is defined in parse_cmd_ln.c as a list of strings (using CMD_LN_STRING_LIST) and thus is stored in an array of pointers (char **). This is because there are typically multiple accumulator directories, because the Forward-Backward part of training (using 'bw') is often run in multiple parts on a cluster of networked machines.

The most common use case for 'norm' is to generate a new set of acoustic model files from accumulator directories alone. It is also possible to specify input mean and variance files to be copied into the new model files in the case where some model parameters were not observed. After validating the command-line arguments, the code checks to see if these files were specified, and reads in the data from them:

#! cplusplus
    if (in_mean_fn != NULL) {
        E_INFO("Selecting unseen density mean parameters from %s\n",
               in_mean_fn);
        if (s3gau_read(in_mean_fn,
                       &in_mean,
                       &n_mgau,
                       &n_gau_stream,
                       &n_gau_density,
                       &veclen) != S3_SUCCESS) {
          E_FATAL_SYSTEM("Couldn't read %s", in_mean_fn);
        }
        ckd_free((void *)veclen);
        veclen = NULL;
    }
    if (in_var_fn != NULL) {
        E_INFO("Selecting unseen density variance parameters from %s\n",
               in_var_fn);
        if (var_is_full) {
            if (s3gau_read_full(in_var_fn,
                           &in_fullvar,
                           &n_mgau,
                           &n_gau_stream,
                           &n_gau_density,
                           &veclen) != S3_SUCCESS) {
                E_FATAL_SYSTEM("Couldn't read %s", in_var_fn);
            }
        }
        else {
            if (s3gau_read(in_var_fn,
                           &in_var,
                           &n_mgau,
                           &n_gau_stream,
                           &n_gau_density,
                           &veclen) != S3_SUCCESS) {
                E_FATAL_SYSTEM("Couldn't read %s", in_var_fn);
            }
        }
        ckd_free((void *)veclen);
        veclen = NULL;
    }

The functions s3gau_read() and s3gau_read_full() are defined in <s3/s3gau_io.h>.

Next, the code iterates over all of the accumulator directories given to it and builds a set of cumulative observation counts from the files in them (some obsolete code has been removed):

#! cplusplus
    for (i = 0; accum_dir[i]; i++) {
        E_INFO("Reading and accumulating counts from %s\n", accum_dir[i]);
        if (out_mixw_fn) {
            rdacc_mixw(accum_dir[i],
                       &mixw_acc, &n_mixw, &n_stream, &n_density);
        }
        if (out_tmat_fn) {
            rdacc_tmat(accum_dir[i],
                       &tmat_acc, &n_tmat, &n_state_pm);
        }
        if (out_mean_fn || out_var_fn) {
            if (var_is_full)
                rdacc_den_full(accum_dir[i],
                               &wt_mean,
                               &wt_fullvar,
                               &pass2var,
                               &dnom,
                               &n_mgau,
                               &n_gau_stream,
                               &n_gau_density,
                               &veclen);
            else
                rdacc_den(accum_dir[i],
                          &wt_mean,
                          &wt_var,
                          &pass2var,
                          &dnom,
                          &n_mgau,
                          &n_gau_stream,
                          &n_gau_density,
                          &veclen);
            if (out_mixw_fn) {
                if (n_stream != n_gau_stream) {
                    E_ERROR("mixw inconsistent w/ densities WRT # "
                            "streams (%u != %u)\n",
                            n_stream, n_gau_stream);
                }
                if (n_density != n_gau_density) {
                    E_ERROR("mixw inconsistent w/ densities WRT # "
                            "den/mix (%u != %u)\n",
                            n_density, n_gau_density);
                }
            }
            else {
                n_stream = n_gau_stream;
                n_density = n_gau_density;
            }
        }
    }

This is accomplished by simply adding together the counts from each directory to produce a running total, which is done using the functions rdacc_mixw(), rdacc_tmat(), and rdacc_den() (or rdacc_den_full() for full covariance matrices). These functions are defined in <s3/s3acc_io.h>. The variables mixw_acc, tmat_acc, wt_mean, wt_var, and wt_fullvar are used to store the cumulative counts. You may or may not remember that the Baum-Welch update formula for maximum-likelihood estimation of the means of a continuous-density HMM is:

#! latex
\begin{displaymath}
\hat\mu_{jk} = \frac{\sum_{t=1}^T \gamma_t(j,k) \vec o_t}{\sum_{t-1}^T \gamma_t(j,k)} 
\end{displaymath}

The SphinxTrain variables

wt_mean[i][j][k]

and

dnom[i][j][k]

correspond exactly to the numerator (note that this is a vector) and the denominator (note that this is a scalar) of this equation. Therefore to do normalization we basically just have to divide

wt_mean

by

dnom

. This is actually done in the file

src/libs/libmodinv/gauden.c

by the function

gauden_norm_wt_mean

. In

norm

, it is called in the following piece of code:

#! cplusplus
    if (wt_mean || wt_var || wt_fullvar) {
	if (out_mean_fn) {
	    E_INFO("Normalizing mean for n_mgau= %u, n_stream= %u, n_density= %u\n",
		   n_mgau, n_stream, n_density);

	    gauden_norm_wt_mean(in_mean, wt_mean, dnom,
				n_mgau, n_stream, n_density, veclen);
	}
	else {
	    if (wt_mean) {
		E_INFO("Ignoring means since -meanfn not specified\n");
	    }
	}

	if (out_var_fn) {
	    if (var_is_full) {
		if (wt_fullvar) {
		    E_INFO("Normalizing fullvar\n");
		    gauden_norm_wt_fullvar(in_fullvar, wt_fullvar, pass2var, dnom,
					   wt_mean,	/* wt_mean now just mean */
					   n_mgau, n_stream, n_density, veclen,
					   cmd_ln_boolean("-tiedvar"));
		}
	    }
	    else {
		if (wt_var) {
		    E_INFO("Normalizing var\n");
		    gauden_norm_wt_var(in_var, wt_var, pass2var, dnom,
				       wt_mean,	/* wt_mean now just mean */
				       n_mgau, n_stream, n_density, veclen,
				       cmd_ln_boolean("-tiedvar"));
		}
	    }
	}
	else {
	    if (wt_var || wt_fullvar) {
		E_INFO("Ignoring variances since -varfn not specified\n");
	    }
	}
    }
    else {
	E_INFO("No means or variances to normalize\n");
    }

For variances, the standard Baum-Welch formula is:

#!latex
\begin{displaymath}
\hat\Sigma_{jk} = \frac{\sum_{t=1}^T \gamma_t(j,k) (\vec o - \vec\mu_{jk})(\vec o - \vec\mu_{jk})^T}{\sum_{t-1}^T \gamma_t(j,k)}
\end{displaymath}

Note that the denominator of this equation is the same as in the mean re-estimation formula, and therefore the SphinxTrain variable

dnom

is used in both re-estimations. In the case of diagonal covariances, the outer product operation in the numerator reduces to a simple element-wise squaring. Therefore the variance can be re-estmated independently in each dimension just like the mean. There is one further wrinkle to do with two possible versions of this formula, which will be discussed in more depth below.

Finally, we write out the newly re-estimated means and variances, which is done with the function

s3gau_write

, in this code:

#! cplusplus
    if (out_mean_fn) {
	if (wt_mean) {
	    if (s3gau_write(out_mean_fn,
			    (const vector_t ***)wt_mean,
			    n_mgau,
			    n_stream,
			    n_density,
			    veclen) != S3_SUCCESS)
		return S3_ERROR;
	    
	    if (out_dcount_fn) {
		if (s3gaudnom_write(out_dcount_fn,
				    dnom,
				    n_mgau,
				    n_stream,
				    n_density) != S3_SUCCESS)
		    return S3_ERROR;
	    }
	}
	else
	    E_WARN("NO reestimated means seen, but -meanfn specified\n");
    }
    else {
	if (wt_mean) {
	    E_INFO("Reestimated means seen, but -meanfn NOT specified\n");
	}
    }
    
    if (out_var_fn) {
	if (var_is_full) {
	    if (wt_fullvar) {
		if (s3gau_write_full(out_var_fn,
				     (const vector_t ****)wt_fullvar,
				     n_mgau,
				     n_stream,
				     n_density,
				     veclen) != S3_SUCCESS)
		    return S3_ERROR;
	    }
	    else
		E_WARN("NO reestimated variances seen, but -varfn specified\n");
	}
	else {
	    if (wt_var) {
		if (s3gau_write(out_var_fn,
				(const vector_t ***)wt_var,
				n_mgau,
				n_stream,
				n_density,
				veclen) != S3_SUCCESS)
		    return S3_ERROR;
	    }
	    else
		E_WARN("NO reestimated variances seen, but -varfn specified\n");
	}
    }
    else {
	if (wt_var) {
	    E_INFO("Reestimated variances seen, but -varfn NOT specified\n");
	}
    }

Writing SphinxTrain programs in Python

The Python modules included with SphinxTrain make it easy to write small scripts that manipulate acoustic models. They can also be used to examine models interactively through the Python shell. We highly recommend installing the matplotlib and ipython extensions to Python, which, in combination, provide a convenient, MATLAB-like environment for manipulating and viewing numerical data.

Let's look at the Python equivalent of the code for 'norm' described above. We'll skip command line parsing since you can do that using the Python getopt module, and just assume that all the accumulator directories live under the directory 'bwaccumdir' in the current directory. We will also assume that the python modules are installed in the 'python' directory under the current directory, which is what SphinxTrain's training setup script will do for you by default.

First, we have to set up the environment and load the necessary modules:

#! python
#!/usr/bin/env python
import os
import sys
sys.path.append('python')
from sphinx import s3gaucnt

In this case the

s3gaucnt

and

s3gau

modules are the only ones we actually need. The first module provides the

sphinx.s3gaucnt.S3GauCntFile

class and some helper functions which we will use to process observation count files, while the second one provides the

sphinx.s3gau.S3GauFile

class which we will use to write out the re-estimated parameter files. Now we will use the

sphinx.s3gaucnt.accumdirs

function to accumulate counts from a list of directories: (if we were dealing with full covariance accumulators, we would use

sphinx.s3gaucnt.accumdirs_full

)

#! python
# Accumulate observation counts from all accumulation directories
gauden = s3gaucnt.accumdirs([os.path.join('bwaccumdir', x) for x in os.listdir('bwaccumdir')])

The return value for this function is a

sphinx.s3gaucnt.S3GauCntFile

object which contains the merged counts for all the items in the list of directories passed to

accumdirs

. We can then obtain the mean, variance, mixture weight, and transition matrix counts from this object, as well as the normalization constant (“dnom”, which corresponds to the denominator of the Baum-Welch update formulae), using a set of accessor functions.

Let's normalize the means first, since they are very simple. The means are obtained using the

getmeans

method, while the normalizer is obtained using

getdnom

. What you actually get is a list of lists of 2-dimensional arrays (of type

numpy.ndarray

). For more information on how to manipulate these, please see the NumPy documentation.

#! python
# Normalize means
outmeans = []
# For each codebook in the counts file and its associated set of normalizers:
for mgau, mgau_dnom in zip(gauden.getmeans(), gauden.getdnom()):
    # Create a list of re-estimated parameters for this codebook and append it to the output array
    outmgau = []
    outmeans.append(outmgau)
    # For each feature stream in this codebook and its associated set of normalizers:
    for feat, feat_dnom in zip(mgau, mgau_dnom):
        # Normalize the parameters (dividing a 2-D array by a 1-D array)
        outmgau.append(feat / feat_dnom)
# Write out the re-estimated means
s3gau.open("means", "wb").writeall(outmeans)

And that's it, seriously. Note how we can use

zip

to ensure that we match up codebooks and features with their corresponding normalizers without ever having to deal with any index variables.

itertools.izip

might be a bit more efficient in this case.

Now we will do the variances. These are only slightly more complicated due to the fact that we have to distinguish between “two-pass” and “one-pass” variance accumulators. You may recall that there are two mathematically equivalent forms of the maximum likelihood estimator of variance:

#! latex
\begin{displaymath}
\sigma^2 = E[(x-\mu)^2] = E[x^2] - \mu^2
\end{displaymath}

In the first case the sufficient statistic is «latex($(x-\mu)^2$)» while in the second case it is «latex($x^2$)» and does not depend on the means at all. In Sphinx speak, the first case is called “two-pass” estimation, and is used when the

-2passvar yes

flag is specified to

bw

. The count file contains a flag which describes which type of variance statistics it contains. This is stored in the

pass2var

attribute of the

sphinx.s3gaucnt.S3GauCntFile

object. So, our variance normalization looks like this:

#! python
# Normalize variances
outvars = []
# For each codebook in the counts file and its associated set of normalizers:
for mgau, mgau_dnom, mgau_mean in zip(gauden.getvars(), gauden.getdnom(), outmeans):
    # Create a list of re-estimated parameters for this codebook and append it to the output array
    outmgau = []
    outvars.append(outmgau)
    # For each feature stream in this codebook and its associated set of normalizers:
    for feat, feat_dnom, feat_mean in zip(mgau, mgau_dnom, mgau_mean):
        if gauden.pass2var:
            # Two pass variance: statistic is (x-\mu)^2, we just have to normalize it
            outmgau.append(feat / feat_dnom)
        else:
            # One pass variance: statistic is x^2, we have to subtract the squared re-estimated mean
            outmgau.append(feat / feat_dnom - feat_mean * feat_mean)
# Write out the re-estimated variances
s3gau.open("variances", "wb").writeall(outvars)

And finally, to normalize the mixture weights and transition matrices, we do… nothing. In fact that is what the original

norm

program does as well. The reason for this is that the normalization constant for these can be computed quickly when the models are loaded, and there are some benefits to storing them in unnormalized form (for one, it allows them to be used as priors for MAP adaptation).

sphinxtrainwalkthrough.txt · Last modified: 2010/02/21 12:28 by admin