File Parsing

igpp.docgen can parse files and define a context that contains the contents as variables.

Supported formats are:

csv: for delimited text

cdf: for CDF data files

json: for JSON formated data

list: for variable lists

pds3: for PDS3 formatted information

csv - Delimited table file parsing (.csv, .tab)

Files with the extension ".csv" or ".tab" are parsed as a delimited table.

Field names are taken from the first record. Comments are lines that begin with "#" and are ignored. The array of records is assigned the name "record" in the defined Velocity context. The number of records can be determined with record.size().

Example Use

cdf - Common Data Format (.cdf)

Files with the extension ".cdf" are parsed as CDF file.

The metadata contained in the CDF is made available as follows:

attributes: The global attributes in the file.

variables: Each variable in the file.

data: Only the variables with a CDF type of "data".

version: The version of the CDF distribution that created the CDF file (note: version.release.increment).

release: The release of the CDF distribution that created the CDF file (note: version.release.increment).

increment: The increment of the CDF distribution that created the CDF file (note: version.release.increment).

encoding: The byte level encoding of the CDF file (for example, SUN_ENCODING, MAC_ENCODING, IBMPC_ENCODING).

pathName: The path and file name for the CDF file.

fileMD5: The MD5 checksum for the CDF file.

Example Use

json - Javascript Object Notation (.json)

Files with the extension ".json" are parsed as Javascript Object Notation.

Supports single valued elements, objects and arrays, but limits the nesting of array to two-levels. That is, an array of arrays, but not an array of arrays in an array. JSON allows each value in an array to be of a different data type. However, igpp.docgen requires all array values be of the same type, so igpp.docgen coerces all array values to be a string type.

Example Use

List - Text file parsing (.txt)

Files with the extension ".txt" are considered a text file with keyword=value assignments.

When text files are parsed lines beginning with "#" are considered comments. All other lines are parsed as a "name=value". If a line does not conform to the "name=value" syntax it is ignored. If a value is enclosed in curly braces {} the value is parsed as a comma separated list.

Lookup Tables

Keyword/Value text files can be used to create simple lookup tables. The value associated with a keyword can be retrieved with a

$context.get(name)

where "context" is the defined context in Velocity and "name" is the keyword to find. "name" can also be a reference to parameter in another context. For example, to retrieve the value from a "target" context using the value for a spacecraft in a "list" context do the following:

$target.get($list.spacecraft)

Example Use

PDS3 Label Parsing (.lbl)

Files with the extension ".lbl" are considered a PDS3 label file.

When a PDS3 label is parsed a variable hierarchy is created which matches the organization of objects in the label. If multiple objects occur at the same level an array is created in the variable tree.

The PDS3 label parser is a Object Definition Language (ODL) parser and is very lenient as to compliance with the PDS3 data model schema. You can use ODL to define nested values using the "OBJECT=name" to nest values.

All OBJECT definitions create a new array in the generated Velocity context. This is true even if there is only one occurrence of an OBJECT definition. In the Velocity template use a "#foreach" directive to process each OBJECT.

Data File

A detached label has the same base name as the data file it describes. Both the label and the data file must be located in the same folder. When the PDS3 label is parsed the file system is search for the corresponding data file. Attributes for the located file are included with in the top level variable for the label. The added attributes are:

PRODUCT_FILE: The file name of the data file.

PRODUCT_FILE_MD5: The MD5 checksum for the data file.

Pointers

PDS3 supports pointers which are named references to a file which are tied to an OBJECT. A pointer has the general form:

^{name}_{class}=({file}, {location})

where {name} is a unique prefix for the object refernce, {class} is the object class (i.e., FILE), {file} is the name of the file and {location} is the location within the file to the start of the object. When a PDS3 label is parsed the class of the pointer object is used for the variable reference with attributes added to the variable to describe the pointer and the referenced file. The attributes included in the variable are:

OBJECT_NAME: The name portion of the pointer.

OBJECT_FILE: The file referenced by the pointer.

OBJECT_FILE_MD5: The MD5 checksum for the file.

OBJECT_LOCATION: The location to the start of the object from the start of the file.

Example:

/*Band A */  
^HFRA_CAL_LEVELS_AGC_TABLE   = ("T2007002_CAL0.TAB", 4) /* Agc */  
OBJECT                       = HFRA_CAL_LEVELS_AGC_TABLE  
   INTERCHANGE_FORMAT        = ASCII  
   ROW_BYTES                 = 140  
   ROWS                      = 4  
   COLUMNS                   = 2  
   DESCRIPTION               = ""  
   OBJECT                    = COLUMN  
	  NAME                   = CALIBRATION_LEVEL  
	  DATA_TYPE              = CHARACTER  
	  START_BYTE             = 1  
	  BYTES                  = 6  
	  DESCRIPTION            = "HFR internal calibration active antenna stimulus."  
   END_OBJECT                = COLUMN  
END_OBJECT                   = HFRA_CAL_LEVELS_AGC_TABLE

The file associated with the TABLE in this example will be referenced as:

$label.TABLE.Product_file

Example Use