Script to perform evaluation experiments. More...

Data Structures
class	spr_eval.EvalConfig
	Handles processing and storing of configuration data. More...

Namespaces
	spr_eval

Functions
def	spr_eval.res_sub
	Replace words. More...

def	spr_eval.send_ext_cmds
	Send extra commands to the recognizer at various stages. More...

def	spr_eval.start_recognizer
	Start the recognizer and read/configure resources according to the specs in the 'SPRaak.ini' file. More...

def	spr_eval.stop_recognizer
	Stop the recognizer. More...

def	spr_eval.process_corpus
	Process the entries in the corpus file. More...

def	spr_eval.main
	Entry point of the script. More...

Detailed Description

Script to perform evaluation experiments.

Synopsis

  spr_eval.py -v [LM/LEX/SEARCH]=<value> <config.ini> <base> <corpus> <obsdir> <suffix> [env=val] ...

Arguments

<config.ini>

File describing one or more configurations (acoustic model, language model, lexicon, ...) for the recognizer

<base>

Just a name, used to create the output files:

<base>.CMD the raw commands as sent to cwr_main
<base>.ERR the error output of cwr_main
<base>.OUT the raw output of cwr_main
<base>.RES a result file extracted from the raw output
<base>.LAT a (word) lattice file (if requested)

<corpus>

Corpus file to use; gives base file names (no leading directory, no suffix) and begin/end times

<obsdir>

Directory where the sample files can be found

<suffix>

Extension to add after the base file names in the corpus file

LM/LEX/...

Select a non default language model, lexicon or set of search options

Extra arguments must have the format 'env=val' and are values that will be added to the list of environment variables.

Config file

The config file holds named parameters which are subdivided into sections. Every parameter must be part of a section and a section is started by a single line with the section name between square brackets. Values of parameters can contain tags of the form ${<env-name>}. These will be substituted by values of the environment variables named by <env-name>. If the environment variable does not exist, the tag will be replaced with the empty string and a warning is given. Here is an overview of all the supported sections and parameters:

   [audio]
     nchan      <nchan>
     dataformat <data-fmt>
     samplefreq <samp-freq>
   [preproc]
     script     <script_fname>
     test_file  <test_fname>
     ext.wav    N01;256
   [phonset]
     CI         <ci_fname>
     CD         <cd_fname>
   [acmod]
     hmm        <hmm_fname>
     mvg        <mvg_fname>
     rmg        <rmg_fname>
     sel        <sel_fname>
     LMO        <log_min_out>
     confidence 1/0
   [default]
     langmod    <default_LM_name>
     acmod      <default_AM_name>
     lexicon    <default_LEX_name>
     max_bw     <default_search_param>
     min_bw     <default_search_param>
     threshold  <default_search_param>
     wlat       <create_word_lattice>
        % 0: no; 1: yes; 2: yes, add phone segmentation; 3: yes, add state segmentation
     mode       CWR
                % CWR: continuous word recognition (default); IWR: isolated word recognition
   [LM:xxx]
     file       <fname>
     type       <FSG/...>
     options    <LM-opt>
     cost_A     <A-value>
     cost_C     <C-value>
   [AM:xxx]
     hmm        <hmm_fname>
     mvg        <mvg_fname>
     rmg        <rmg_fname>
     sel        <sel_fname>
     LMO        <log_min_out>
     confidence 1/0
   [LEX:xxx]
     file       <fname>
     unwind     <unwind-opt>
   [SEARCH:am,lex,lm]
     max_bw     <max-beam-width>
     min_bw     [min-beam-width](0)
     threshold  <prob_threshold>
     frac_wend  [frac_wend](0.6)
     lm_cache   [lm_cache_sz](-1)
     lmc_cache  [lmc_cache_sz](-1)
     add_min_frac [add_min_frac](1.0)

The sections describing the language model, the acoustic model, the lexicon and the search parameters can have multiple versions. The version that is active will depend on the LM, AM, LEX and SEARCH command line arguments or on the langmod, acmod and lexicon parameters in the default section.

Example

Here's an real-world example configuration file. It shows the use of the replacement tags and it has multiple versions of the lexicon and language model sections.

 [audio]
   nchan      1
   dataformat SHORT
   samplefreq 16000
 [preproc]
   script     PREPROCnew modWSJ1_CMUv2.4/acmod.preproc
   test_file  /dev/null
 [phonset]
   CI         phon.ci
   CD         modWSJ1_CMUv2.3/acmod.cd
 [acmod]
   hmm        modWSJ1_CMUv2.4/acmod.hmm
   mvg        modWSJ1_CMUv2.4/acmod.mvg
   sel        modWSJ1_CMUv2.4/acmod.sel
   rmg        m0.005;new;bmp=0;
   LMO        0.0
 [default]
   basedir    /users/spraak/krisdm/werk/exp/dialect/wsj_train
   langmod    t5k
   lexicon    5k
   max_bw     ${BW}
   min_bw     0
   threshold  ${THR}
   omit       <s> </s> <sil>
   wlat       0
 
 [LEX:5k]
   file       ../wlist5c_nvp.cmu+s.dic
   unwind     add_in_front=;add_at_rear=;sil_word=<sil>;
 [LEX:20k]
   file       ../wlist20o_nvp.cmu+s.dic
   unwind     add_in_front=;add_at_rear=;sil_word=<sil>;
 
 [LM:b5k]
   file       /users/spraak/spchdata/wsj/lm/bcb05cnp.v73.lm.gz
   options    FWD1;check_lvl=0;CLOSED;ext=wsj/lm/lm.ext
   cost_A     ${LMA}
   cost_C     ${LMC}
 [LM:t5k]
   file       /users/spraak/spchdata/wsj/lm/tcb05cnp.v73.lm.gz
   options    FWD1;check_lvl=0;CLOSED;ext=wsj/lm/lm.ext
   cost_A     ${LMA}
   cost_C     ${LMC}
 [LM:b20k]
   file       /users/spraak/spchdata/wsj/lm/bcb20onp.v73.lm.gz
   options    FWD1;check_lvl=0;ext=wsj/lm/lm.ext
   cost_A     ${LMA}
   cost_C     ${LMC}
 [LM:t20k]
   file       /users/spraak/spchdata/wsj/lm/tcb20onp.v73.lm.gz
   options    FWD1;check_lvl=0;ext=wsj/lm/lm.ext
   cost_A     ${LMA}
   cost_C     ${LMC}

The above file could for example be used with the following command line call:

  $ spr_eval.py -v LEX=5k -v LM=t5k WSJ1_cmu_v2.ini test_new test.cor ~spchdata/wsj/sam_16k/wsj0 wv1 BW=50000 THR=81 LMA=4.0 LMC=-10

The values of the LEX and LM arguments are used to activate the correct version of lexicon and language model parameters. The values at the end of the call will be used to replace tags in the config file.

History

07/2005 - KD: Script added to the hmm75 repository
05/2008 - JR: Script converted to python and added to the SPRAAK repository

Author: Kris Demuynck (gawk code); Jan Roelens (python code)

Data Structures

Namespaces

Functions

Detailed Description