RAW Product Format

The RAW product is a collection of all raw ingest data acquired during a run of a single task (volume scan).

The raw product is a single file into which many ingest files have been incorporated.

For hybrid tasks, individual raw products are made from each of the individual tasks that make up the overall scan. Optionally the RAW product can be made in separate files on a sweep-by-sweep basis.

Raw product files are blocked into 6144-byte records, which match the record length used if and when the files are eventually written to tape. Mimicking the tape structure on disk permits error recovery to be built directly into the raw data format. For all other types of products, if a tape I/O error occurs within the product's records on tape, then the entire product is lost. For raw archive this would be too great a penalty, since an entire volume scan is at stake. The blocking scheme permits partial error recovery while still maintaining a one-to-one mapping between disk and tape formats.

The records are for IRIS interpretation only and do not refer to any operating system file records.

The first 6144 byte record of a RAW product holds the product_hdr (See product_hdr Structure) structure.

The RAW product begins with a product_hdr like all the other types of products. The only difference is the zero padding out to 6144 bytes.

The next record holds the ingest_header (see ingest_header Structure) structure for the volume scan that supplied the data. This structure is zero-padded to fill the entire second record.

All subsequent records hold the actual data, and each record begins with the raw_prod_bhdr structure. See 1-byte Velocity Format (DB_VEL).

Records hold data from one sweep. The sweep number is in each record's header. The data for the sweep is the concatenation of all the data from as many records as pertain to that sweep. When data for a sweep ends short of the 6144 byte record size, the remainder of that record is padded with zeros. Each of these sweep data sets begins with the ingest_data_header (See ingest_data_header Structure) structures for each data type that was recorded. The same number of headers are found in the beginning of a sweep's data as there are data types acquired during the sweep. The list of data types recorded is specified by the data collection mask in the task_dsp_info in the task_configuration in the ingest_header. The rays of data are immediately after the headers.

Rays are ordered within a sweep by the same ordering sequence used for the ingest data file pointer table. If a ray is missing from the ingest file, a zero-length ray is inserted into the product file as a placeholder. The number of compressed rays in the file is equal to the product of the number of data types recorded and the number of angles sampled. This is true even if data were not acquired at some of those angles. Within a ray, the recorded data are ordered by increasing data type number. See the task_dsp_info structure for a definition of the data type numbers. See task_dsp_info Structure.

Record #1 { <product_hdr> 0,0,0... }
Record #2 { <ingest_header> 0,0,0... }
Record #3 { <raw_prod_bhdr> <ingest_data_header(s)> Data... }
Record #4 { <raw_prod_bhdr> Data... }
. . .
. . .
Record #N { <raw_prod_bhdr> Data 0... }
Record #N+1 { <raw_prod_bhdr> <ingest_data_header(s)> Data... }
Record #N+2 { <raw_prod_bhdr> Data... }
. . .
. . .
Record #M { <raw_prod_bhdr> Data 0... }