Requirements
SPRAAK has the following dependencies:
- Python (version 2.5 or higher)
- All scripts in SPRAAK are written in Python2
- A Posix-compliant thread library
- To speed up the processing, some components in SPRAAK can make use of multiple threads. Currently, SPRAAK uses the POSIX standard.
The functionality of SPRAAK can be further extended when the link libraries and header files for the following components are available:
- python
- When not only the Python excutable but also the Python header files and library are installed, Python functionality can be made available from with the SPRAAK system (e.g. a Python signal pre-processing block, a Python lattice post-processing block, ...).
- zlib
- This library allows reading and writing gzip-compressed files.
- libbz2
- This library allows reading and writing bzip2-compressed files.
- readline
- This library allows reading allow and interactive command line in spr_cwr_main
- gcrypt
- This library allows reading and writing encrypted files.
For building the SPRAAK binaries, SPRAAK also relies on:
- A C99 compliant compiler
- Note
- None of the current compilers (gcc, visual-C, ...) are 100% C99 compliant. However, since SPRAAK avoids the more 'difficult' C99 constructs, this is not a problem.
-
SPRAAK works optimally in combination with gcc and clang/llvm (even on Windows, we use gcc; see below on how to install a working gcc-based compilation enviroment on Windows).
- Scons (version 1.1 or higher)
- Scons is a Python based make-system.
For building the documentation (optional), SPRAAK relies on:
- Doxygen (1.5 or higher)
- Doxygen is used to create the documentation.
- Graphviz (dot)
- Doxygen cat use 'dot' to draw dependency graphs. If 'dot' cannot be found, the dependency graphs will be omitted from the documentation.
- pdflatex
- Pdflatex can be used for creating the manuals in pdf format.
Linux (and more in general *nix) system
On all *nix distributions on which SPRAAK was tested, all dependencies could be solved with standard packages that came with the *nix distribution. See the documentation of the distribution on how to install additional packages.
Windows
For compiling SPRAAK on Windows, we advise the MSYS2 build enviroment (good for building SPRAAK as native Windows applications, but not a complete *nix on Windows enviroment). Optionally, the Cygwin environment (a close to complete *nix enviroment on Windows) can be used to compile the SPRAAK applications as Cygwin applications (untested) and/or for deploying SPRAAK on Windows machines (either using the native Windows binaries or the Cygwin binaries).
- Installing the MSYS2 64 bits build environment:
-
Download and install 'msys2-x86_64-*.exe' from www.msys2.org
-
Follow the instructions on www.msys2.org to update all components.
-
Install the extra components needed to build SPRAAK:
pacman -S scons doxygen mingw-w64-x86_64-clang mingw-w64-x86_64-zlib mingw-w64-x86_64-bzip2 mingw-w64-x86_64-winpthreads mingw-w64-x86_64-readline mingw-w64-x86_64-termcap
-
Optional: install extra components to facilitate developing and testing in msys2
pacman -S vim openssh man diffutils tcsh
# switch back to dumb path completion if you find the smart completion behaves randomly
pacman -R bash-completion
-
To compile SPRAAK, start the 'MSYS2 MingW 64-bit' environment from the 'MSYS2 64bit' environment
- Installing the MSYS2 32 bits build environment:
-
Download and install 'msys2-i686-*.exe' from www.msys2.org
-
Follow the instructions on www.msys2.org to update all components.
-
Install the extra components needed to build SPRAAK:
pacman -S scons doxygen mingw-w64-i686-clang mingw-w64-i686-zlib mingw-w64-i686-bzip2 mingw-w64-i686-winpthreads mingw-w64-i686-readline mingw-w64-i686-termcap
-
Optional: install extra components to facilitate developing and testing in msys2
pacman -S vim openssh man diffutils tcsh
# switch back to dumb path completion if you find the smart completion behaves randomly
pacman -R bash-completion
-
To compile SPRAAK, start the 'MSYS2 MingW 32-bit' environment from the 'MSYS2 32bit' environment
- Note
- Both Graphviz (dot) and pdflatex are missing in the msys2 environment, and hence SPRAAK's documentation will be somewhat less rich when build on Windows.
-
Due to a bug in Python/scons, the compile process may try to write temporary files in 'C:\Windows', and due to the more strict security features in Windows 7/8/10, this directory is no longer writable for everyone.
Solution1: Start the MSYS2 shell in 'administrator' mode (this does not work if one wants to install SPRAAK on a network drive since the administrator cannot access network drives).
Solution2: Use cygwin (MSYS2 does not map ownership by default – see the MSYS2 documentation for details on how to change this behavior) to change the ownership of '/cygdrive/c/Windows' chown ${USER}:${USER} /cygdrive/c/Windows
OSX / MacOS
SPRAAK can be compiled on OSX. The required extra tools can be readily obtained via macports and other open software package managers such as fink. Hereunder, the different steps when using macports are described.
- Xcode
- Install Xcode via the App-store and run Xcode (the last step is required to complete the installation).
- Xcode command line tools
- Install the command line tools by opening a command window and running the following command If this fails, you can also try to install the command line tools from the developer website: https://developer.apple.com/download/more/
- Macports
- Download and install the macports package for your version of OSX from one of these sources
-
https://guide.macports.org/
-
https://github.com/macports/macports-base/releases/
- Update and install packages
- Update and install the packages needed to compile SPRAAK by opening a command terminal and issuing the following commands:
sudo port selfupdate
sudo port upgrade outdated
sudo port install scons zlib libedit
# zlib and libedit are also needed by scons, so you don't even need to request for their installation
If you intend to use python interactively, then you may also have to install the following package (see the warnings when installing python2.7 if this is needed or not) sudo port install py27-readline
To create the documentation, you need to install the following extra packages sudo port install doxygen graphviz
- Install SPRAAK
- Install SPRAAK using scons as explained in Configuring & installing
- Note
- installing on a mounted disk may cause clang (the compiler used by OSX) to fail, so first try installing SPRAAK on a native OSX volume before filling a bug report
Other/new platforms
SPRAAK depend on standard libraries and open-source tools only, so migration to other platforms should be possible. Hereunder, we list some of the more challenging tools to obtain.
- Scons
- Scons can be donwloaded from http://www.scons.org. Choose the appropriate download type in the right column. If you choose the tarball, you can proceed as follows:
tar zxvf scons-*.tar.gz
cd scons-*
Read the README file. It explains the installation procedure and prerequisites.
As a quickstart, you can type python setup.py install --prefix=$SPRAAK_BASE_DIR/scons_base
to install scons in the directory referred to above as $SPRAAK_BASE_DIR/scons_base
- Posix threads
- The Posix thread library is available by default on all unix systems.
On windows, you can use the pthreads-win32 library: see sourceware.org/pthreads-win32/
- Doxygen
- Doxygen can be downloaded directly from www.doxygen.org
- Graphviz (dot)
- Doxygen cat use 'dot' to draw dependency graphs. If 'dot' cannot be found, the dependency graphs will be omitted from the documentation. For Linux system, we advise to use the Graphviz package that commes with the distribution. Alternatively, one can download Graphviz from http://www.graphviz.org/ At the time of writing no Cygwin package for Graphviz was available. It should be possible to manually build the Graphviz tools from source in Cygwin, but your mileage may vary.
- pdflatex
- Pdflatex can be used for creating the manuals in pdf format and can be found in lexlive.
Configuring & installing
SPRAAK uses 'scons' (a Python build-tool) the compile and install the software.
All user configurable settings (location of the different files, compiler options, ...) are grouped in the file 'config.py' in the root directory of the SPRAAK package. See the comments in config.py and the section Package structure for more info.
The default configuration of SPRAAK allows multiple versions (32 and 64 bits versions; different operating systems and/or processors; debug, release and profile versions; ...) to be installed and maintained in parallel on a single shared file system. Hence, the default directory structure differs somewhat from what is typical in packages ment for local installation only. See Package structure for more details.
To see the all possible build options, run
Calling
will make a build using the default option (CONFIG=release, COMPILER=gcc, ARCH=auto, EXPORT=developer). This results in a library and executables that are aimed for final release (don't contain debug, profile, ... information). At the same time, the header files and the documentation are aimed at developers, i.e. they expose everything.
Compiling SPRAAK without the documentation can be done with:
By default, the resulting SPRAAK binaries will run on recent CPUs (less than 10 years old) only. If older CPUs must be supported, add the 'ARCH=compatible' option. On the other hand, by targetting a specific CPU (adding option 'ARCH=native'), somewhat faster binaries can be obtained at the expense of compatibility.
On Linux, the clang/llvm compiler results in somewhat faster (and smaller) binaries, especially on modern CPUs (with support for 256bit vectors or better).
scons COMPILER=clang ARCH=native install
In order to be able to run the programs, users have to adjust their login scripts so that the PATH environment variable includes the location of the SPRAAK executables. On systems which do not support locating dynamic library files relative w.r.t. the executables, the LD_LIBRARY_PATH environment variable (or equivalent) must be adjusted to include the location of the SPRAAK library as well.
- Note
- The spr_version.py script can set/adjust all necassary environment variables on supported platforms.
Testing the software
The examples
directory contains a series of example experiments that may be used to test if the software behaves correctly. See Introduction and Background for more details on the different examples and the expected outcome.
Package structure
The source files for the SPRAAK package are organized as follow:
- ./src/lib/
- The C-code that will be compiled to form the SPRAAK library. The documentation and some additional information for constructing headers files is embedded in the C-code.
- ./src/prog/
- The C-code that will be compiled to form the SPRAAK executables. The documentation is embedded in the C-code.
- ./src/doc/
- Free-standing documentation (i.e. documentation that has no 1-to-1 link to any of files in ./src/{lib,prog}).
- ./scripts/spr_parser.py
- The parser to extract header files and documentation for the C source files and to create the object oriented glue code for each class.
- ./scripts/spr_*
- High level Python scripts in SPRAAK framework.
- ./scripts/spraak/
- A library of Python code, used by the SPRAAK scripts.
- ./SConstruct
- The master build-file (using scons).
- ./config.py
- Build-file containing all user configurable options.
- ./sysdep.py
- Build-file containing the code to detect system dependent features, bugs, libraries, ...
Other files, related to but not an integrated part of SPRAAK, can be found in:
- ./src_ext/
- Extra C-code for interfacing SPRAAK (library and/or file IO) with other environments such as Matlab.
- ./scripts/alien/
- High level scripts that have been found to be useful in combination with SPRAAK
Upon building (using the default settings), the following directory structure will be created:
- ./build/
- All items derived by the build process: header files, documentation in intermediate form, object files, libraries, ...
- ./build/include/<lvl>/spraak/
- The include files with all items at export level <lvl> or higher.
- ./build/include/<lvl>/spraak_classdef/
- The class specific definition derived by the parser with all items at export level <lvl> or higher.
- ./build/include/<arch>/spraak/
- Architecture dependent include files.
- ./build/parse_tmp/
- All information extracted by the parser (spr_parser.py) stored in raw format.
- ./build/doc_tmp/<lvl>/
- The documentation in Doxygen format as extracted by the parser; contains only the items at export level <lvl> or higher.
- ./build/doc/<lvl>/html/
- The documentation in html format as derived by Doxygen.
- ./build/doc/<lvl>/latex/
- The documentation in pdflatex format as derived by Doxygen.
- ./build/<arch>/obj/
- Object files created during the build process.
- ./build/<arch>/lib/
- The library file.
- ./build/<arch>/bin/
- The executables.
- ./export
- All installed items (scripts, executables, libraries, header files and documentation).
- Note
- The default setup expect that the package maintainer only installs (
scons install
) the desired export level <lvl> for his environment. If one wants to install multiple export levels (or make sure only a given export level can be installed), the file config.py must be adjusted!
- ./export/bin/spr_*
- The (Python) scripts.
- ./export/bin/spraak/
- The Python library used by the scripts.
- ./export/bin/<arch>/
- The executables.
- ./export/lib/<arch>
- The library file.
- ./export/include/
- Include files.
- ./export/doc/html/
- Documentation in html format.
The export level <lvl> can have the following values:
- pub_hi
- header files, documentation, library aimed at users.
- pub_lo
- header files, documentation, library aimed at programmers.
- devel
- header files, documentation, library aimed at developers.
The <arch> infix combines information about the operating system, CPU and configuration (debug, release, ...) into a single name.
In order to facilitate package management, SPRAAK also allows for setting symbolic links from the 'export' directories to some final destination directories (e.g. /usr/bin, /usr/lib and /usr/include). This way, all files from the SPRAAK package can be kept in a single directory while users still can have access to the full functionality without having the expand their PATH, LD_LIBRARY_PATH, ...). Furthermore, when SPRAAK is removed, upgraded, ... the worst that can happen is some dangling links in the /usr/bin, /usr/lib, and /usr/include/ directories. See the comments in config.py for more info.
config.py
22 not_windows = (spraak.os!=
"Windows")
26 spraak.bindir = spraak.prefix+
'/bin'
28 spraak.bindir_arch = spraak.prefix+
'/bin/'+spraak.arch
30 spraak.libdir = (spraak.prefix+
'/lib' if(not_windows)
else spraak.bindir)
32 spraak.libdir_arch = (spraak.prefix+
'/lib/'+spraak.arch if(not_windows)
else spraak.bindir_arch)
34 spraak.includedir = spraak.prefix+
'/include'
36 spraak.docdir_html = spraak.prefix+
'/doc/html'
37 spraak.docdir_pdf = spraak.prefix+
'/doc/pdf'
39 spraak.datadir = spraak.prefix+
'/data'
45 spraak.bindir_link =
None
46 spraak.libdir_link =
None
47 spraak.includedir_link =
None
49 spraak.symlink_support = not_windows
55 spraak.builddir = spraak.prefix+
'/build'
82 spraak.python_h =
None
86 spraak.zlib_h =
"auto"
98 spraak.gcrypt_h =
None
101 spraak.arpack =
"auto"
102 spraak.arpack_h =
"auto"
105 spraak.readline =
"auto"
106 spraak.readline_h =
"auto"
109 spraak.editline =
"auto"
110 spraak.editline_h =
"auto"
117 def select_best_match(opt_dict,spraak):
123 sel = [spraak.os,spraak.cpu,spraak.cc,str(spraak.cc_ver[0]),spraak.config];
126 for opt_str,opt_val
in opt_dict.iteritems():
128 for ndx,val
in enumerate(opt_str.split()):
133 if((opt_val!=
None)
and (cnt<best_cnt)):
136 if(best_opt ==
None):
137 print "ERROR No match found for %s %s %s %s %s"%(spraak.os,spraak.cpu,spraak.cc,str(spraak.cc_ver[0]),spraak.config)
138 spraak.signal_error = 1;
158 cflags_opt = {
"* * gcc * release" :
"-O2",
159 "* * gcc * debug" :
"-O2 -g",
160 "* * gcc * regression" :
"-O2 -g -DSPR_REGRESSION=1",
161 "* * gcc * profile" :
"-O2 -pg",
162 "* * gcc * coverage" :
"-O1 -ftest-coverage -fprofile-arcs",
163 "* x86_64 gcc * release" :
"-O3 -fno-keep-static-consts -maccumulate-outgoing-args -ffast-math -mieee-fp -funroll-loops -fpeel-loops -funswitch-loops -finline-limit=250 --param max-unroll-times=4",
164 "* x86_64 gcc * debug" :
"-g -O3 -maccumulate-outgoing-args -ffast-math -mieee-fp -funroll-loops -fpeel-loops -funswitch-loops -finline-limit=250 --param max-unroll-times=4",
165 "* i686 gcc * release" :
"-O3 -fno-keep-static-consts -maccumulate-outgoing-args -fstrict-aliasing -fomit-frame-pointer -ffast-math -mieee-fp -funroll-loops -fpeel-loops -funswitch-loops -finline-limit=250 --param max-unroll-times=4",
166 "* i686 gcc * debug" :
"-g -O3 -maccumulate-outgoing-args -fstrict-aliasing -fomit-frame-pointer -ffast-math -mieee-fp -funroll-loops -fpeel-loops -funswitch-loops -finline-limit=250 --param max-unroll-times=4",
167 "Windows i686 gcc * release" :
"-O3 -fno-tree-vectorize -fno-keep-static-consts -maccumulate-outgoing-args -fstrict-aliasing -fomit-frame-pointer -ffast-math -mieee-fp -funroll-loops -fpeel-loops -funswitch-loops -finline-limit=250 --param max-unroll-times=4",
168 "Windows i686 gcc * debug" :
"-g -O3 -fno-tree-vectorize -maccumulate-outgoing-args -fstrict-aliasing -fomit-frame-pointer -ffast-math -mieee-fp -funroll-loops -fpeel-loops -funswitch-loops -finline-limit=250 --param max-unroll-times=4",
169 "OSX * gcc * release" :
"-O3 -ffast-math -mieee-fp",
170 "OSX * gcc * debug" :
"-g -O3 -ffast-math -mieee-fp",
171 "* * clang * release" :
"-O3 -ffast-math",
172 "* * clang * debug" :
"-g -O3 -ffast-math",
173 "* x86_64 clang * release" :
"-O3 -ffast-math -mieee-fp",
174 "* x86_64 clang * debug" :
"-g -O3 -ffast-math -mieee-fp",
175 "* i686 clang * release" :
"-O3 -ffast-math -mieee-fp",
176 "* i686 clang * debug" :
"-g -O3 -ffast-math -mieee-fp",
179 cpu_x86_nbits = {
"i686":
"-m32 ",
"x86_64":
"-m64 "}.get(spraak.cpu,
"");
180 cpu_x86_nbits = {
"m32":
"-m32 ",
"m64":
"-m64 "}.get(env[
"CROSS"],cpu_x86_nbits);
181 arch_gcc_i686 = cpu_x86_nbits + {
"auto":
"-march=core2 -mtune=core2 -malign-double ",
"native":
"-march=native -mtune=native -malign-double ",
"compatible":
""}[env[
"ARCH"]];
182 arch_clang_i686 = cpu_x86_nbits + {
"auto":
"-march=core2 -mtune=core2 ",
"native":
"-march=native -mtune=native ",
"compatible":
""}[env[
"ARCH"]];
183 arch_x86 = cpu_x86_nbits + {
"auto":
"-march=core2 -msse3 -mtune=native ",
"native":
"-march=native -mtune=native ",
"compatible":
""}[env[
"ARCH"]];
185 cflags_cfg = {
"* * * * *" :
"",
186 "* * gcc * *" :
"-std=gnu99 -fpic -fsigned-char",
187 "* i686 gcc * *" :
"-std=gnu99 -fpic " + arch_gcc_i686 +
"-W -Wall -Wno-uninitialized -Wno-unused-value -fsigned-char -fasynchronous-unwind-tables",
188 "Windows i686 gcc * *" :
"-std=gnu99 " + arch_gcc_i686 +
"-W -Wall -Wno-uninitialized -Wno-unused-value -fsigned-char",
189 "* x86_64 gcc * *" :
"-std=gnu99 -fpic " + arch_x86 +
"-W -Wall -Wno-uninitialized -Wno-unused-value -fsigned-char -fasynchronous-unwind-tables",
190 "Windows x86_64 gcc * *" :
"-std=gnu99 " + arch_x86 +
"-W -Wall -Wno-uninitialized -Wno-unused-value -fsigned-char",
191 "* i686 clang * *" :
"-std=c99 -fpic " + arch_clang_i686 +
"-W -Wall -Wno-incompatible-pointer-types-discards-qualifiers -fsigned-char -fasynchronous-unwind-tables",
192 "Windows i686 clang * *" :
"-std=c99 " + arch_clang_i686 +
"-W -Wall -Wno-incompatible-pointer-types-discards-qualifiers -fsigned-char",
193 "* x86_64 clang * *" :
"-std=c99 -fpic " + arch_x86 +
"-W -Wall -Wno-incompatible-pointer-types-discards-qualifiers -fsigned-char -fasynchronous-unwind-tables",
194 "Windows x86_64 clang * *" :
"-std=c99 " + arch_x86 +
"-W -Wall -Wno-incompatible-pointer-types-discards-qualifiers -fsigned-char",
196 cflags_shobj ={
"* * * * *" :
"-DSPR_INCL_LVL=-1",
197 "Linux * * * *" :
"-DSPR_INCL_LVL=-1 -fvisibility=hidden -fno-common",
198 "OSX * * * *" :
"-DSPR_INCL_LVL=-1 -fno-common",
199 "Windows * * * *" :
"-DSPR_INCL_LVL=-1 -fno-common",
201 spraak.cflags = select_best_match(cflags_cfg,spraak)+
' '+select_best_match(cflags_opt,spraak)
202 spraak.shcflags = select_best_match(cflags_shobj,spraak)+
' '+spraak.cflags
205 ldlibs = {
"Linux * * * *" :
"m dl rt pthread",
206 "Linux * gcc * *" :
"m dl rt pthread gcc_s",
207 "OSX * * * *" :
"m dl pthread",
208 "Windows * * * *" :
"m wsock32 winmm",
210 ldpath = {
"* * * * *" :
"",
211 "Windows * gcc * *" :
"extern/lib",
213 ldcpph = {
"* * * * *" :
"",
214 "Windows * gcc * *" :
"extern/include",
216 spraak.ldlibs = select_best_match(ldlibs,spraak)
217 spraak.ldpath = select_best_match(ldpath,spraak)
218 spraak.ldcpph = select_best_match(ldcpph,spraak)
222 ldflags = {
"Linux * * * *" : cpu_x86_nbits +
"-rdynamic -Wl,-z,origin -Wl,-rpath,'$$ORIGIN/lib'",
223 "Linux * * * profile" : cpu_x86_nbits +
"-pg -rdynamic -Wl,-z,origin -Wl,-rpath,'$$ORIGIN/lib'",
224 "Linux * * * coverage" : cpu_x86_nbits +
"-ftest-coverage -fprofile-arcs -rdynamic -Wl,-z,origin -Wl,-rpath,'$$ORIGIN/lib'",
225 "OSX * * * *" : cpu_x86_nbits +
"-rdynamic",
226 "Windows * * * *" : cpu_x86_nbits +
"-static-libgcc -Wl,--disable-stdcall-fixup -Wl,--enable-auto-import",
229 spraak.ldflags = select_best_match(ldflags,spraak)
230 ldlibflags = {
"* * * * *" :
"",
231 "Linux * * * *" : cpu_x86_nbits +
"-rdynamic -Wl,-O -Wl,--enable-new-dtags -Wl,--as-needed",
232 "OSX * * * *" : cpu_x86_nbits +
"-dynamiclib -install_name @executable_path/lib/libspraak.dylib",
233 "Windows * * * *" : cpu_x86_nbits +
"-static-libgcc -Wl,--out-implib=spraak.dll.a -Wl,--enable-stdcall-fixup -Wl,--export-all-symbols -Wl,--enable-auto-import",
235 spraak.ldlibflags = select_best_match(ldlibflags,spraak)
238 cppflags = {
"* * * * *" :
"",
239 "* * gcc * *" :
"-D_XOPEN_SOURCE=600 -D_ISOC99_SOURCE -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D_REENTRANT",
240 "* * clang * *" :
"-D_XOPEN_SOURCE=600 -D_ISOC99_SOURCE -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D_REENTRANT",
242 spraak.cppflags = select_best_match(cppflags,spraak)
249 bugs = {
"* * * * *" : [],
250 "* x86_64 * * *" : ([
"x86_twin48_18"] if(env[
"ARCH"]==
"compatible")
else [])
252 for bug
in select_best_match(bugs,spraak):
253 setattr(spraak,
'sysdep_bug_'+bug,
True);
278 spraak.sysdep_bug_sphere_shortpack_v0 =
True;