spr_train.py File Reference

Script to create and train acoustic models. More...

Namespaces
	spr_train

Functions
def	spr_train.main
	Main function. More...

Detailed Description

Script to create and train acoustic models.

Synopsis

  spr_train.py <training-description-file>

Arguments

<training-description-file>

File describing the different steps and iterations required for the creation and/or training of an acoustic model.

Training description

The description file contains python code that will be executed in it's own environment. That environment gives access to two python objects: config and trainer (see spr_pylib::train::master::create_new_trainer() ). The config object will hold all configuration data while the trainer object will be used to perform the different training steps. Setting a configuration value is done by assigning values to the relevant attribute of the config object. The first call to a method from the trainer object will trigger a sanity check of the configuration values. If during this check some values are found missing the script will abort with an error stating what value is missing.

For more information on the various attributes available on the config object see spr_pylib::train::master::MasterConfig .

For more information on the various methods available on the trainer object see spr_pylib::train::master::MasterTrainer .

Distributed computing using ssh

Remote processes will be started using ssh. For this you need to make sure that the account that will run the training scripts can login on the remote nodes without user interaction. For more info see the man pages on ssh and public-private key authentication and the ssh-add command. It is also required that all the remote nodes share the same filesystem. All data communication happens via the shared filesystem. The following steps are taken to lauch a remote process:

• ssh login to the remote node. Make sure everything is configured so that no user interaction is required during the logon.
• The working directory is set so as to be the same as the working directory as set when lauching the master training script. This avoids the need to remap paths to files given to the various SPRAAK commands.
• Optionally extra commands can be run to setup environment variables. (See remote_env_script )
• Be friendly to other users of the node: nice -n 19
• The actual command is run. The script waits for it to finish and checks the return code to see if the command was successful.

Crash recovery

The training scripts keep track of what steps of the training were executed correctly. When the training script is launched again after it has been interupted, it will skip any training steps that were already completed during a previous run. The progress information is stored in a log file with a name of the form <exp-name>_recovery.log. This file is located in the <metadir> directory. It is important to remove this file if one wants to redo a complete run from start to finish. This is for example needed after changes to the config file or any of the starting input files.

Selective rollback

By editing the recovery log file it is possible to selectively redo some steps of the training without having to start from the beginning. This way it can be possible to skip expensive training steps if they don't have to be redone. To do a rollback you have to edit the recovery log by removing lines from the end of the file up to the point where you want to resume the training. It's only possible to restart a trainig from a recovery point. These are marked in the recovery log file. If you try to perform a rollback to a point where there is no recovery point, the results may not be correct or the script may even abort due to missing files. There should be recovery points after all expensive steps.

Note1

Instead of specifying the occupance, one can also specify the target average mixture size mx_sz by setting key oc* to sz.<mx_sz> .

Note2

The optional Viterbi aligment at the end of each major iteration can be forced by setting the key dovit to 1.

Note3

At the end of each major iteration, a test can be done by setting the key tst to the name of the test-script. The test-script is then called with the following arguments:

     <exp> <model_name> "<ph_ci> <ph_cd>" "<preproc> <ppe>" <topt[1]> ... <topt[8]>

The extra arguments <topt[1...8]> can be defined as <key>=<values> pairs as well

Example

Here's an example config file used during testing:

 config.exp = "RM"
 config.log = "%s.LOG" % config.exp
 config.ldir = "ldir"
 config.sdir = "sdir"
 config.mname = "mod{exp}.{ITER}/{name}{iter}"
 config.preproc = "PREPROC"
 config.mida_np = 17
 config.mida_opt = "-gc MMI -gf LIN -VC -1 -niter 10 -eps 5e-4 -greedy 'Gg' -Fopt '/'"
 config.ph_ci = "phon.ci"
 config.ph_cd = "phon.cd"
 config.ph_arcd = "phon.arcd"
 config.ph_spec = "\#"
 config.questions = "questions"
 config.seg = "RM.seg"
 config.cor = "RM.cor"
 config.dic = "RM_man.dic"
 config.unwind = "'add_in_front=\#;add_between=[/\#];add_at_rear=\#;sent_context=\#\#;'"
 config.obsdir = "/data/jroelens/RM_SAM_DATA"
 config.suffix = "sam"
 config.pass_split_cnt = 1
 config.local_paths.append("/volume1/")
 config.remote_env_script = ". ../../SETENV"
 \#config.host_info = [(None, 2), ("spchcl06", 2)]
 config.host_info = [(None, 2)]
 config.metadir = "%s_PROGRESS" % config.exp
 
 trainer.tied(niter = 1, tng = 1024)
 trainer.cdtree(niter = 1, dtmc = 512, dtlt = 1024)
 trainer.fvg(niter = 1)

See Also

spr_pylib.train.master

History

07/2005 - KD

script added to the hmm75 repository

10/2007 - JR

script converted to python and added to the SPRAAK repository

Author

Kris Demuynck (gawk code)

Jan Roelens (python code)