Overview

This document explains some of the motivation and background behind various
design decisions that went into defining SDIF.

Why Isn't SDIF IFF Compatible?

In early versions of SDIF, we claimed compatibility with the IFF standard,
the parent of several important standards AIFF
and RIFF.
However, we have decided to drop strict IFF compatibility because of problems
that arose in trying to ensure 64-bit alignment of
all data types.

Here's the issue: All IFF files consist of one large chunk, which must begin
with some header fields. AIFF files are a "FORM" chunk, whose header
contains these 3 fields:

  • "FORM" (4 bytes),
  • a 4 byte size count for the entire file or stream (4 bytes), and
  • the type of the file, e.g., "AIFF" or "SDIF" (4 bytes)

This adds up to 12 bytes, which makes all subsequent chunks not 16-byte (64
bit) aligned. We could have solved this problem with a mandatory chunk whose
length is 4 more than a multiple of 16 bytes, but that seemed too much like
a kludge.

Other IFF chunks besides FORM have similar problems; all have manadatory "headers"
whose sizes are not multiples of 16 bytes.

Another problem with IFF is that a 4 byte size count is not big enough for
certain extremely large sets of SDIF data.

The SDIF standard still follows the "spirit" of IFF, with a series
of frames each with an identifying 4-byte frame type and size count; the only
practical difference is the lack of an opening "FORM" chunk.

We don't know of any commonly available utilities or libraries for manipulating
IFF files in general, so it seems like we haven't given anything up with this
decision.

How To Embed SDIF in IFF Files

Here's a proposal for embedding SDIF data in IFF files such as AIFF files.
Embedded SDIF data can be no longer than about 2 gigabytes, because IFF chunks
contained a signed 32-bit count of the data size.

Embedded SDIF data would require a special IFF form chunk to be a "wrapper"
around the entire SDIF block:

ChunkID
char[4]
'FORM', as required by IFF

ChunkSize
int32
The size, in bytes, of the entire embedded SDIF data, plus this chunk,
not including the "FORM" ChunkID or this ChunkSize field.

FormType
char[4]
'SDIF' (We need to register this form type!)

PaddingChunkID
char[4]
'SPAD'

PaddingChunkSize
int32
Either 0, 2, 4, or 6

PaddingChunkData
char[n]
0 to 6 bytes of null characters, depending on PaddingChunkSize

SDIFChunkID
char[4]
'SDIF'

SDIFChunkSize
int32
The size, in bytes, of the embedded SDIF data

SDIFData
data
The SDIF data

This structure is a legal IFF chunk and can therefore be embedded inside other
IFF chunks. It consists of an enclosing FORM chunk that includes two subchunks.
The first subchunk is a padding chunk to ensure that the SDIF data will be aligned
on an 8-byte boundary. The second subchunk contains the SDIF data to be embedded,
wrapped in a legal IFF chunk.

The number of padding bytes depends on the size of the data that precedes this
chunk in the IFF file (which must always be a multiple of 2 bytes, per the IFF
standard). For example, suppose the preceeding portion of the file is 200 bytes.
Those 200 bytes, plus the 28 non-padding bytes in the above wrapper, is 228,
which is 4 more than a multiple of 8, so there would need to be 4 padding bytes
to make the SDIF data begin on an 8-byte boundary.

Note that changing the contents of the earlier portion of an IFF file may require
changing the number of padding bytes.

We welcome suggestions about better
ways to do this.

Why Is the Opening Frame's Format Different
From Other Frames?

The opening frame is the only frame that does not have a time tag, a stream
ID, and matrices. Why this nonuniformity? The SDIFSpecVersion and SDIFStandardTypesVersion
could be data elements in a matrices in a frame at time tag minus infinity.


Two reasons:

  • If a future version of SDIF has a new format for frame headers or matrix
    headers, it will be impossible to find the SDIFSpecVersion that indicates
    that a given file uses the new format.
  • Space efficiency: the opening frame would be 48 bytes long instead of only
    16.

64-bit Alignment

  • We want to have the option to work with 64-bit data.
  • In order to do that in an efficient way, we have to require that all 64-bit
    data items are aligned to 64-bit in respect to the beginning of the file.
    This is especially important for working with memory mapping of the complete
    file, for example with the 'mmap' utilities in UNIX. We made tests for SGI
    O2s and DEC Alphas that showed that you have to align to 64-bit in memory
    in order to work with 64-bit floats (For the test program mail
    to Rolf Woehrmann).
  • We want to have the time information in the data frames always in 64-bit
    floats. So the beginnings of the data frames have to be always aligned to
    64-bits. In order to achieve that we have to require possible byte padding
    at the end of each matrix data. This further allows to have only specific
    matrixes in a single data frame in 64 bit.

Why Allow Optional Columns In Matrices?

There are many cases where it would make sense to add extra columns to a matrix.
For example, the Lemur project does analysis for and synthesis of "bandwidth
enhanced" sinusoids with the usual amplitude, frequency, and phase fields
plus a "bandwidth" field indicating the spectral width or "noisiness"
of the partial. Another example is a measure of the "importance" or
perceptual salience of each of a set of partials.

SDIF supports this in a way that allows programs
that don't understand these extra columns to read and process the information
they do understand without disturbing the extra information: Extra columns must
always appear after the required columns.