Crash due to a computer crash

Given that acoustic model training can take multiple days of CPU time, it's not fun if your experiment gets trashed inadvertently. SPRAAK has a logging mechanism that will allow to restart an experiment that crashed due to some external problem (e.g. machine going down). For experiment 'exp' a recovery logfile 'exp_recovery.log' is created logging recovery points that were passed successfully. An experiment will continue running starting of at the last recovery point by simply restarting the experiment.

Crash due to a human mistake

The above described crash recovery procedure may inadvertently get in the way when you made an error in the .config file !! as the from the system's point of view nothing may have gone wrong way beyond the point that your errors were effective.

So if you want to restart an experiment from scratch, be sure to remove all output generated by the system so far and especially the recovery logfile 'exp_recovery.log' !!

It will help you if you develop for yourself a filenaming mechanism that easily distinguish input files (exp.config, exp.ini) from generated files (e.g. exp_.log, exp_models, exp_.OUT, ...).