edu.cmu.sphinx.decoder.search
Class Token

java.lang.Object
  edu.cmu.sphinx.decoder.search.Token

All Implemented Interfaces:

Scoreable, Data, java.io.Serializable

public class Token
extends java.lang.Object
implements Scoreable

Represents a single state in the recognition trellis. Subclasses of a token are used to represent the various emitting state.

All scores are maintained in LogMath log base

See Also:

Serialized Form

Field Summary

private static int	curCount
private int	frameNumber
private static int	lastCount
private int	location
private float	logAcousticScore
private float	logInsertionScore
private float	logLanguageScore
private float	logTotalScore
private float	logWorkingScore
private Data	myData
private static java.text.DecimalFormat	numFmt
private Token	predecessor
private static java.text.DecimalFormat	scoreFmt
private SearchState	searchState
private java.util.HashMap<java.lang.String,java.lang.Object>	tokenProps A collection of arbitrary properties assigned to this token.

Fields inherited from interface edu.cmu.sphinx.decoder.scorer.Scoreable

COMPARATOR

Constructor Summary

Token(SearchState state, int frameNumber)
Creates the initial token with the given word history depth

Token(Token predecessor, float logTotalScore, float logAcousticScore, float logInsertionScore, float logLanguageScore)
Creates a Token with the given acoustic and language scores and predecessor.

Token(Token predecessor, SearchState state, float logTotalScore, float logInsertionScore, float logLanguageScore, int frameNumber)
Internal constructor for a token.

Method Summary

float	calculateScore(Data feature) Calculates a score against the given feature.
void	dumpTokenPath() dumps a branch of tokens
void	dumpTokenPath(boolean includeHMMStates) dumps a branch of tokens
float	getAcousticScore() Returns the acoustic score for this token (in logMath log base).
Data	getData() Returns the feature for this Token.
int	getFrameNumber() Returns the frame number for this token.
float	getInsertionScore() Returns the insertion score associated with this token.
float	getLanguageScore() Returns the language score associated with this token
int	getLocation() Returns the location of this Token in the ActiveList.
protected static java.text.DecimalFormat	getNumberFormat() Return the DecimalFormat object for formatting the print out of numbers
Token	getPredecessor() Returns the predecessor for this token, or null if this token has no predecessors
float	getScore() Returns the score for the token.
protected static java.text.DecimalFormat	getScoreFormat() Return the DecimalFormat object for formatting the print out of scores.
SearchState	getSearchState() Returns the SearchState associated with this token
java.util.Map<java.lang.String,java.lang.Object>	getTokenProps() Returns the application object
Word	getWord() Returns the word of this Token, the search state is a WordSearchState.
java.lang.String	getWordPath() Returns the string of words for this token, with embedded silences
java.lang.String	getWordPath(boolean wantFiller, boolean wantPronunciations) Returns the string of words leading up to this token.
java.lang.String	getWordPathNoFiller() Returns the string of words for this token, with no embedded filler words
java.lang.String	getWordUnitPath() Returns the string of words and units for this token, with embedded silences.
float	getWorkingScore() Gets the working score.
boolean	isEmitting() Determines if this token is associated with an emitting state.
boolean	isFinal() Determines if this token is associated with a final SentenceHMM state.
boolean	isWord() Determines if this token marks the end of a word
float	normalizeScore(float maxLogScore) Normalizes a previously calculated score
void	setData(Data data) Sets the feature for this Token.
void	setLocation(int location) Sets the location of this Token in the ActiveList.
void	setScore(float logScore) Sets the score for this token
void	setWorkingScore(float logScore) Sets the working score for this token
static void	showCount() Shows the token count
java.lang.String	toString() Retrieves the string representation of this object
boolean	validate() Determines if this branch is valid

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

curCount

private static int curCount

lastCount

private static int lastCount

scoreFmt

private static final java.text.DecimalFormat scoreFmt

numFmt

private static final java.text.DecimalFormat numFmt

predecessor

private final Token predecessor

logLanguageScore

private final float logLanguageScore

logTotalScore

private float logTotalScore

logInsertionScore

private float logInsertionScore

logAcousticScore

private float logAcousticScore

logWorkingScore

private float logWorkingScore

searchState

private final SearchState searchState

location

private int location

frameNumber

private final int frameNumber

myData

private Data myData

tokenProps

private java.util.HashMap<java.lang.String,java.lang.Object> tokenProps

A collection of arbitrary properties assigned to this token. This field becomes lazy initialized to reduce memory footprint.

Constructor Detail

Token

public Token(Token predecessor,
             SearchState state,
             float logTotalScore,
             float logInsertionScore,
             float logLanguageScore,
             int frameNumber)

Internal constructor for a token. Used by classes Token, CombineToken, ParallelToken

Parameters:

predecessor - the predecessor for this token

state - the SentenceHMMState associated with this token

logTotalScore - the total entry score for this token (in LogMath log base)

logLanguageScore - the language score associated with this token (in LogMath log base)

frameNumber - the frame number associated with this token

Token

public Token(SearchState state,
             int frameNumber)

Creates the initial token with the given word history depth

Parameters:

state - the SearchState associated with this token

frameNumber - the frame number for this token

Token

public Token(Token predecessor,
             float logTotalScore,
             float logAcousticScore,
             float logInsertionScore,
             float logLanguageScore)

Creates a Token with the given acoustic and language scores and predecessor.

Parameters:

logAcousticScore - the log acoustic score

logLanguageScore - the log language score

predecessor - the predecessor Token

Method Detail

getPredecessor

public Token getPredecessor()

Returns the predecessor for this token, or null if this token has no predecessors

Returns:

the predecessor

getFrameNumber

public int getFrameNumber()

Returns the frame number for this token. Note that for tokens that are associated with non-emitting states, the frame number represents the next frame number. For emitting states, the frame number represents the current frame number.

Specified by:

getFrameNumber in interface Scoreable

Returns:

the frame number for this token

setData

public void setData(Data data)

Sets the feature for this Token.

Parameters:

data -

getData

public Data getData()

Returns the feature for this Token.

Returns:

the feature for this Token

getScore

public float getScore()

Returns the score for the token. The score is a combination of language and acoustic scores

Specified by:

getScore in interface Scoreable

Returns:

the score of this frame (in logMath log base)

calculateScore

public float calculateScore(Data feature)

Calculates a score against the given feature. The score can be retrieved with get score. The token will keep a reference to the scored feature-vector.

Specified by:

calculateScore in interface Scoreable

Parameters:

feature - the feature to be scored

Returns:

the score for the feature

normalizeScore

public float normalizeScore(float maxLogScore)

Normalizes a previously calculated score

Specified by:

normalizeScore in interface Scoreable

Parameters:

maxLogScore - the score to normalize this score with

Returns:

the normalized score

getWorkingScore

public float getWorkingScore()

Gets the working score. The working score is used to maintain non-final scores during the search. Some search algorithms such as bushderby use the working score

Returns:

the working score (in logMath log base)

setWorkingScore

public void setWorkingScore(float logScore)

Sets the working score for this token

Parameters:

logScore - the working score (in logMath log base)

setScore

public void setScore(float logScore)

Sets the score for this token

Parameters:

logScore - the new score for the token (in logMath log base)

getLanguageScore

public float getLanguageScore()

Returns the language score associated with this token

Returns:

the language score (in logMath log base)

getInsertionScore

public float getInsertionScore()

Returns the insertion score associated with this token. Insertion score is the score of the transition between states. It might be transition score from the acoustic model, phone insertion score or word insertion probability from the linguist.

Returns:

the language score (in logMath log base)

getAcousticScore

public float getAcousticScore()

Returns the acoustic score for this token (in logMath log base). Acoustic score is a sum of frame GMM.

Returns:

score

getSearchState

public SearchState getSearchState()

Returns the SearchState associated with this token

Returns:

the searchState

isEmitting

public boolean isEmitting()

Determines if this token is associated with an emitting state. An emitting state is a state that can be scored acoustically.

Returns:

true if this token is associated with an emitting state

isFinal

public boolean isFinal()

Determines if this token is associated with a final SentenceHMM state.

Returns:

true if this token is associated with a final state

isWord

public boolean isWord()

Determines if this token marks the end of a word

Returns:

true if this token marks the end of a word

toString

public java.lang.String toString()

Retrieves the string representation of this object

Overrides:

toString in class java.lang.Object

Returns:

the string representation of this object

dumpTokenPath

public void dumpTokenPath()

dumps a branch of tokens

dumpTokenPath

public void dumpTokenPath(boolean includeHMMStates)

dumps a branch of tokens

Parameters:

includeHMMStates - if true include all sentence hmm states

getWordPath

public java.lang.String getWordPath(boolean wantFiller,
                                    boolean wantPronunciations)

Returns the string of words leading up to this token.

Parameters:

wantFiller - if true, filler words are added

wantPronunciations - if true append [ phoneme phoneme ... ] after each word

Returns:

the word path

getWordPathNoFiller

public java.lang.String getWordPathNoFiller()

Returns the string of words for this token, with no embedded filler words

Returns:

the string of words

getWordPath

public java.lang.String getWordPath()

Returns the string of words for this token, with embedded silences

Returns:

the string of words

getWordUnitPath

public java.lang.String getWordUnitPath()

Returns the string of words and units for this token, with embedded silences.

Returns:

the string of words and units

getWord

public Word getWord()

Returns the word of this Token, the search state is a WordSearchState. If the search state is not a WordSearchState, return null.

Returns:

the word of this Token, or null if this is not a word token

showCount

public static void showCount()

Shows the token count

getLocation

public final int getLocation()

Returns the location of this Token in the ActiveList. In the HeapActiveList implementation, it is the index of the Token in the array backing the heap.

Returns:

the location of this Token in the ActiveList

setLocation

public final void setLocation(int location)

Sets the location of this Token in the ActiveList.

Parameters:

location - the location of this Token

validate

public boolean validate()

Determines if this branch is valid

Returns:

true if the token and its predecessors are valid

getScoreFormat

protected static java.text.DecimalFormat getScoreFormat()

Return the DecimalFormat object for formatting the print out of scores.

Returns:

the DecimalFormat object for formatting score print outs

getNumberFormat

protected static java.text.DecimalFormat getNumberFormat()

Return the DecimalFormat object for formatting the print out of numbers

Returns:

the DecimalFormat object for formatting number print outs

getTokenProps

public java.util.Map<java.lang.String,java.lang.Object> getTokenProps()

Returns the application object

Returns:

the application object