Data IO in SPRAAK

Shortcuts

• From physical storage to data content
• Specifying a stream of data (file)
• Loose ends and details
• Some practical examples

From physical storage to data content

In order to cope with the wide variety of how data can be stored, SPRAAK distinguishes 4 layers when accessing data.

• The protocol layer: this layer copes with how the data can be physically accessed, for example a disk on file, or an audio device, or a file on an ftp-server.
• The coder1 and coder2 layers: these layers cope with encryption, compression or any other form of streaming transformation of the data. There can be multiple coders stacked on top of each other and coders can work either on the data only (coder2 layer) or on both the data and the meta data (coder1 layer). The information for seting up the coder2 layer is stored as meta data, i.e. as part of the header.
• The header layer: this layer handles the meta data (header information) that describes the data at hand. Certain headers automatically imply a certain encoding of the data, and hence imply the use of certain (de)coders in one of the coder layers.

When writing, the (meta) data flowsthrough the layers depicted below from left to right, i.e. the meta data (the header) is first sent to the header layer (which encodes the header information and sends the data through unmodified). If the header specifies (by means of the 'COMPRESS' key) that the data must be compressed or otherwise encoded, the data will be first sent to 'coder2'. The header followed by the data is then sent to the coders in the 'coder1' layer, and finaly this data stream is written to the physical media in the protocol layer.

[physical storage] <--> protocol  <-->  coder1  <-->  header  <-->  coder2  <--> [data in memory]
                                                               `---------------> [meta data / header info]

When reading, the (meta) data follows the reverse path (from left to right), i.e. the data is first read form the physical media in the protocol layer, then sent through the (de)coders (in the 'coder1' layer) in the reverse order. Next, the header layer will extract the header information after which it will send through all data unmodified. If the 'COMPRESS' key is set, the the data will be sent trough an additional (de)coder (in the 'coder2' layer).

SPRAAK contains default implementations to handle a variety of protocols, coders and header formats. Support for new protocols, coders and header formats can be added by user programs as well.

By default, SPRAAK knows the following protocols:

file

disk based files (or named unix pipes)

sock

stream oriented sockets (TCP-protocol)

dev

devices that need specific handling, e.g. audio IO

cmd

(unix) commands passing data through nameless (unix) pipes

mem

a memory based temporary file system

txt

the text specified as the file name (or as parameter)

The raw data read using one of the above protocols can be sent to one or more coders. SPRAAK supports by default the following coders:

gzip

GNU-zip compression and decompression

bzip

bzip2 compression and decompression

gpg

GnuPG encryption and decryption

ssp

spraak stream processing

buf

extra buffering, particulary usefull for communication with high latency protocols (e.g. an external command that processes data in large chunks) or protocols that can only buffer small amounts of data themselves (e.g. audio devices)

sphere

sphere shorten decompression

htk

an HTK feature file

flac

free lossless audio codec (FLAC) compression and decompression

SPRAAK supports the following header formats (with implied encoding of the data) by default:

spraak

A SPRAAK-header, i.e. an ascii-header containing key-value pairs ended with a line containing a single '#' character, followed by the data in binary or ascii format. See The SPRAAK header format

key

A key-file. Provides backward compatibility with the old ESAT-software stack. The format is similary to the SPRAAK-file format (an ascii-header containing key-value pairs ended with a line consisting of one or more '#' characters only, followed by the data in binary or ascii format). However, the keys and values differ substantially. See The key header format

sphere

A NIST sphere-file (SPeech HEader REsources file format). Allows uncompressed data and different styles for compression of audio-data (note: only the shorten decompression algorithm is supported in SPRAAK right now).

htk

A HTK feature file

wav

A RIFF wav-file. Only a limited set of sub-types is supported.

flac

A flac header.

nohdr

Raw data (no header). The meta data (header information) can be specified as part of the file name. See Specifying a stream of data (file).

ehdr

Raw data with an external header. The external header is a file containing a SPRAAK-header (or a key-header). The file containing the external header can be specified as extra argument for the 'ehdr' header specifier (see Specifying a stream of data (file) and Some practical examples for details) or is searched for automatically. The automatic mode only searches in the same directory as the data file. First the extension of the data file is removed and a .ddf (data descriptor file) extension is added. If such a file exist, it is assumed to contain the header info for that specific data file. If such a file does not exist, the header info is assumed to be stored in a file named '.ddf' (again in the same directory as the data file). This second option allows one to define a common header for all files in a directory. When not specified and when possible (no compression, a simple binary type for the data elements), the main size related key (DIM1) is derived automatically based on the size of the data file.

Specifying a stream of data (file)

A stream can be specified in several ways:

• As a simple disk based file name:
  • my_dir/fname.ext
• A quoted string as a simple disk based file name:
  • "my_dir/fname\\twith\\ttabs.ext"
  The use of a quoted string allows the use of escape characters for any problematic character.
• A special file name:

  stdin

  Standard input.

  stdout

  Standard input.

  stderr

  Standard error.

• A simple file name preceded by zero or one protocol specifiers, zero or more coder specifiers, and zero or one header specifiers. The protocol, coder or header specifiers may be followed by optional parameters between square brackets.
  Some examples:
  • ehdr:my_dir/fname.raw
  • ehdr[my_files_fmt.ddf]:my_dir/fname.raw
  • gzip:my_dir/fname.gz
  • gpg[user=spch]:bzip[9]:file:my_dir/fname.bz2
  The (de)coders are applied in the given order when writing and in the reverse order when reading. There may be at maximum one protocol specifier, and if given, the protocol specifier must be the first specifier. There may be at maximum one header specifier, and if given, the header specifier must be the last specifier. The simple file name may again be quoted.
• A set of keyword=value; pairs surrounded by curly brackets:
  • {protocol=file;header=spr;fname=my_dir/fname.ext;}
  • {coder=gzip;fname=my_dir/fname.gz;}
  • {coder=gpg[user=spch];coder=bzip[9];protocol=file;fname="my_dir/fname.bz2";}
  The keywords can be any of the following:

  protocol

  Specify the protocol; this keyword may occur only once

  coder

  Specify the (next) coder; coders are are applied in the given order when writing and in the reverse order when reading.

  fname

  Specify the file name; this keyword may occur only once

  header

  Specify the header format. When the header format is 'ehdr', the location of the external header can be specified using a subsequent 'param=...;' specifier.

  key

  Specify a key:value pair – the colon (':') separates the key and value part

  rmkey

  Remove a key

  The value can be a simple string ending at the terminating semi-colon (';') or a quoted string (allowing the use of escape characters for any problematic character).
  The protocol, coder or header specifiers may be followed by optional parameters between square brackets.
• As a concatenation of multiple files using the construct {+[CAT]+<file1>+<file2>+...+} where <file1>, <file2>, ... can be any of the above constructs. The key specifications (see the '{ }' construct above) are not reset when opening the next file, i.e. all previous 'key=...' and 'rmkey=...' specifications, including those specified for the previous files apply to the current file.
• A merger of multiple files using the construct {+[MERGE]+<file1>+<file2>+...+} where <file1>, <file2>, ... can be any of the above constructs. The key specifications (see the '{ }' construct above) are not reset when opening the next file. How (and if) the 'merge' operation works depends on the content of the first stream.

More details on what can be done with each protocol, coder or header can be found in the documentation of the spr_stream_<cph>_open() function, with <cph> being on of the support coders, protocols or header formats. Some coders (e.g. sphere and flac) are terminating coders, i.e. one cannot have an other coder on top of those.

Most (if not all coders) are auto-detected when reading the file using their magic cookie, making it unnecessary to specify the coders. When writing the file, some coders are auto-detected using the file name extension (.gz, .bz2).

Loose ends and details

Specifying & overwriting keys for I/O

Keys can be specified in three ways:

1. As header in the file (when reading a file).
2. As parameters to the spr_stream_open() function. This is the primary source of keys when writing a file. When reading a file, these keys are used as initial values. Duplicate keys found in the file will overwrite those specified in the keyset.
3. As extra specifiers in the '{...}' construct for specifying the file name. These have the highest priority, i.e. they overwrite keys given in the file or given as parameter.

Back or forward slash and Windows

On windows, the slash ('/') directory seperators are automatically converted to back-slashes ('\') for disk based files. In other words, it is best to use the www (or unix) convention of seperating directories using the slash ('/') character since this will allow transparant porting to web-based or unix-based applications.

Debugging

The stream system can also trace all open files if desired, allowing one to check non-closed streams at the end of the program. This mode is enabled by either giving the '–check_stream' command option or by setting the 'SPR_STREAM_CHECK' environment variable.

Some practical examples

Reading headerless files:

• 'ehdr:my_dir/file.feat'
  Read from a headerless file; the header information must be stored in my_dir/file.ddf or my_dir/.ddf
• 'ehdr[my_other_dir/std.ddf]:my_dir/file.feat'
  Read from a headerless file; the header information is read from my_other_dir/std.ddf
• '{header=nohdr;key=FORMAT:ASCII;key=DATA:TRACK;key=TYPE:I32;key=DIM2:1;fname=stdin;}'
  Read raw data in ASCII format from 'stdin'; all essential keys must be specified.

Writing headerless files:

• 'ehdr:my_dir/file.feat'
  Write a headerless file; use spr_copy -H to extract the matching header.
• '{header=nohdr;key=FORMAT:ASCII;fname=stdout;}'
  Send raw data in ASCII format to 'stdout'.
• '{header=spraak;key=FORMAT:BIN01;fname=stdout;}'
  Send raw data in little-endian format to 'stdout'.