The Configuration Section

The configuration section, described in detail in the "ViPER-GT" manual, contains data typing information for a viper file. This is in the form of a list of descriptor types and the associated attribute information. For a detailed description of the configuration section, refer to The Guide to Authoring Ground Truth with ViPER-GT.

The Data Section

Data is stored as a list of records associated with a media file. Each record represents a contiguous appearance of a descriptor instance in a media file. Type, name and identification number uniquely identifies descriptors. Each descriptor instance record has a contiguous frame span, over which the instance is considered valid; note that if a descriptor is occluded or otherwise marked as invalid for a period of time, it is broken into two records with the same identification number and disjoint, non-adjacent frame spans.

Each record contains a (possibly empty) list of attribute/value pairs. The attributes are as defined in the configuration section of the viper file. If an attribute is not listed, it is set to the default value for the frame span of the record. If it takes no value for some span, it is marked as NULL for that span.

Note that while the older .gtf format only allows one media file to be described with each data file (that ViPER-GT can operate on keyframe lists is not reflected in the file), the XML format is arranged in <sourcefile> elements that allow multiple media file metadata to be stored in a single viper file. It is up to the programmer to decide how to use a multisource viper file; in VideoMiner,

The FILE Information Descriptor

It is recommended that all ground truth contain a FILE Information descriptor. This descriptor can include such information as number of frames, frame rate, and image size. If these attributes are included, then the viper tools can use the information during evaluation and to open the media files automatically when opened with viper-gt.

Specifics of the XML Format

A viper XML file is divided into two sections: config and data. The XML schema for each is required. For more information, see the schemata themselves.

Specifics of the GTF Format

descriptor-type descriptor-name id start-frame:end-frame
    attribute1 : "attribute-value"
    attribute2 : "attribute-value"
    ...
    attributeN : "attribute-value"

Following assumptions are made with regards to GTF format:

1. Attribute names, types and values must be contained on the same line within a data record. A new line will indicate the start of a new attribute or record.

2. Any blank line or the remainder of any line containing a "//" is a comment and will be ignored.

3. Any line beginning with #include "filename" will be used to include the quoted file.

4. The current reserved words include: FILE, CONTENT, OBJECT, default and static.

5. Unknown attribute types, and their values, will be ignored.
6. The GTF file extension is .gtf.

Configurations Section

FILE Information
    SOURCE : svalue [default CNN]
    COMMENT: svalue
    NCLIPS : dvalue [default 14]
    FORMAT : lvalue [MPEG MJPEG AVI MOV]
    FRAME-HEIGHT : dvalue [default 320]
    FRAME-WIDTH : dvalue [default 480]

<descriptor name="Information" type="FILE">
    <attribute name="SOURCE" type="svalue">
        <default>
            <data:svalue value="CNN"/>
        </default>
    </attribute>
    <attribute name="COMMENT" type="svalue"/>
    <attribute name="NCLIPS" type="dvalue">
        <default>
            <data:dvalue value="14"/>
        </default>
    </attribute>
    <attribute name="FORMAT" type="lvalue">
        <data:lvalue-possibles>
            <data:lvalue-enum value="MPEG"/>
            <data:lvalue-enum value="MJPEG"/>
            <data:lvalue-enum value="AVI"/>
            <data:lvalue-enum value="MOV"/>
        </ data:lvalue-possibles >
    </attribute>
    <attribute name="FRAME-HEIGHT" type="dvalue">
        <default>
            <data:dvalue value="320"/>
        </default>
    </attribute>
    <attribute name="FRAME-WIDTH" type="dvalue">
        <default>
            <data:dvalue value="480"/>
        </default>
    </attribute>
</descriptor>

CONTENT Shot-Change
    TYPE : lvalue [CUT FADE-IN FADE-OUT DISSOLVE]

<descriptor name="Shot-Change" type="CONTENT">
    <attribute name="TYPE" type="lvalue">
        <data:lvalue-possibles>
            <data:lvalue-enum value="CUT"/>
            <data:lvalue-enum value="FADE-IN"/>
            <data:lvalue-enum value="FADE-OUT"/>
            <data:lvalue-enum value="DISSOLVE"/>
        </ data:lvalue-possibles >
    </attribute>
</descriptor>

OBJECT Person
    NAME: svalue [static]
    GENDER : lvalue [MALE FEMALE] [default MALE] [static]
    AGE : dvalue [default 0] [static]
    HASHAIR : bvalue [default TRUE] [static]
    INCONVERSATION: relation [PERSON] [static]
    PERCENTVISABLE : fvalue
    NOSE : point
    HEAD : bbox

<descriptor name="Person" type="OBJECT">
    <attribute name="NAME" type="svalue" dynamic="false"/>
    <attribute name="GENDER" type="lvalue" dynamic="false">
        <data:lvalue-possibles>
            <data:lvalue-enum value="MALE"/>
            <data:lvalue-enum value="FEMALE"/>
        </ data:lvalue-possibles >
        <default>
            <data:lvalue value="MALE"/>
        </default>
    </attribute>
    <attribute name="AGE" type="dvalue" dynamic="false">
        <default>
            <data:dvalue value="0"/>
        </default>
    </attribute>
    <attribute name="HASHAIR" type="bvalue" dynamic="false">
        <default>
            <data:bvalue value="true"/>
        </default>
    </attribute>
    <attribute name="INCONVERSATION" type="relation" dynamic="false">
        <data:relation-target value="PERSON"/>
    </attribute>
    <attribute name="PERCENTVISIBLE" type="fvalue"/>
    <attribute name="NOSE" type="point"/>
    <attribute name="HEAD" type="bbox"/>
</descriptor>

Data Section

CONTENT Shot-Change 45 101:108
    TYPE : "DISSOLVE"

<content name=" Shot-Change" id="45" framespan="101:108">
    <attribute name="TYPE">
        <data:lvalue value=""DISSOLVE"/>
    </attribute>
</content>

OBJECT ObjectDesc ID I:I+M
    attribute1 : "attribute-value1,1" , -, "attribute-value1,M"
    attribute2 : "attribute-value2,1" , -, "attribute-value2,M"
    -
    attributeN : "attribute-valueN,1" , -, "attribute-valueN,M"

attribute1 : "k1*(attribute-value1,1)", \.\.\.
attribute2 : "k2*(attribute-value2,1)", \.\.\.
-
attributeN : "kn*(attribute-valueN,1)", \.\.\.

OBJECT Person 22 110:114
    NAME : "JOHN JONES"
    GENDER : "MALE"
    AGE : "24"
    HASHAIR : "TRUE"
    INCONVERSATION: "2 3 45"
    PERCENTVISABLE : 5*("100")
    NOSE : "14 20", 3*("15 20"), "15 21"
    HEAD : "1 1 30 40", 3*("1 1 32 23"), "2 2 33 34"

<object name="Person" id="22" framespan="110:114">
    <attribute name="NAME">
        <data:svalue value="JOHN JONES" />
    </attribute>
    <attribute name="GENDER">
        <data:lvalue value="MALE"/>
    </attribute>
    <attribute name="AGE">
        <data:dvalue value="24"/>
    </attribute>
    <attribute name="HASHAIR">
        <data:bvalue value="TRUE"/>
    </attribute>
    <attribute name="INCONVERSATION">
        <data:relation value="2 3 45"/>
    </attribute>
    <attribute name="PERCENTVISABLE">
        <data:fvalue value="100" span="5"/>
    </attribute>
    <attribute name="NOSE">
        <data:point x="14" y="20"/>
        <data:point x="15" y="20" span="3"/>
        <data:point x="15" y="21"/>
    </attribute>
    <attribute name="HEAD">
        <data:bbox x="1" y="1" width="30" height="40"/>
        <data:bbox x="1" y="1" width="32" height="23" span="3"/>
        <data:bbox x="2" y="2" width="33" height="34"/>
    </attribute>
</object>

Creating an Instance of ViperData

Here is a simple java example of accessing ViPER data:

ViperData data = new ViperDataImpl();
Config fileInfo
    = data.createConfig(Config.FILE, "Information");
fileInfo.createAttrConfig("Framerate", "fvalue", false,
                          new Integer(29.997), null);
Sourcefile f1 = data.createSourcefile("sample.mpg");
Descriptor first = f1.createDescriptor(fileInfo);
first.getAttribute("Framerate")
     .setAttrValue (new Integer(25));
System.out.println ("Framerate is: " +
                    first.getAttribute("Framerate").getAttrValue());

The above code creates a new instance of a ViperData object. A ViperData object is analogous to the entirety of a single ViPER XML file. It then creates the configuration for a File descriptor with the name "Information". As described above, a descriptor with this name and type is important to our tools. It then adds a single "dvalue", or integer, attribute to the descriptor, and then creates an instance of it. Note that it must be attached to a media source file, although that file may have no name.

Reading a File into the API

Here is a sample of code that reads in viper data from a file:

FileInputStream inFile = new FileInputStream ("gt.xml");
Element documentEl = DocumentBuilderFactory.newInstance()
                                           .newDocumentBuilder()
                                           .parse(inFile)
                                           .getDocumentElement();
inFile.close();
ViperData newFile = ViperParser.parseDoc (documentEl);

The above code opens the file "gt.xml" for parsing, parses it into the XML DOM using the javax.xml package, and then extracts the ViperData using viper.api.impl.ViperParser. The Xerces parser from xml.apache.org will also work.

Saving to a File

The current implementation does not support saving to files, only reading and manipulation of the data.

Conversion

As mentioned above and in the other documents, ViPER data files come in two format: a plain text format and an XML format. Both are usable in ViPER-PE, but ViPER-GT requires use of import and export to access data in the older format. While the XML format is the recommended one, it may be simpler to use the old GTF file format for your software; some of the scripts listed in this document still make use of the older format. To address the problem, the gtf2xml and xml2gtf shell scripts, which invoke the viper.descriptors.Convert main method, convert to and from the two video metadata formats.

However, since the XML makes explicit references to the names of the video files it represent(s), it is often prudent to add a FILE Information descriptor to an old format GTF file before running gtf2xml. Convert requires this descriptor to apply the proper names in the sourcefile tag of the XML files. ViPER treats FILE Information descriptors as special descriptors, taking into account NUMFRAMES and FRAME-SIZE attributes, although the only necessary attribute for conversion is the SOURCEFILES attribute.

FILE Information
    SOURCEDIR: svalue
    SOURCEFILES: svalue
    SOURCETYPE: svalue [SEQUENCE FRAMES]
    NUMFRAMES: dvalue

The source directory attribute, SOURCEDIR, provides the base for the relative path names specified in the SOURCEFILES attribute. The SOURCEFILES attribute is a list mapping frames to files. In the case of a single MPEG, a possible value would be "1 vid.mpeg", while in the case of a list of jpeg files it would read "1 a.jpg 2 b.jpg 3 c.jpg".

Note that the number is the start frame of the media file specified in the following token. Currently, the GTF parser will fail on file names that contain white space characters.

Aggregation

Often, it is quicker and easier to compare not just one file to another via viper-pe, but to instead evaluate software on a whole group of files, such as a set of keyframes, pages of a document, or other related media. While ViPER-GT can currently handle only lists of keyframes, the new XML format allows sets of any media type. For an example of this, see the Case Studies document.

There are a few caveats for aggregation. One is to make sure that each input file refers to the correct file name. The software is case sensitive, so it is important to make sure that the source media names are each the same case in all locations, which may not occur in windows or when transferring files via CD-R. The sourcefile information is stored in different locations in the two viper data formats. In the classic, more human readable ground truth format, it is stored in the FILE Information descriptor as described both above and in the appendix for the ViPER-GT manual. For the newer XML format, the source media file name is found in the sourcefile elementÂ's filename attribute.

Modification

As listed in the table of command line arguments above, the Converter utility allows several different types of modification. While many modifications should be able to be performed with the ViPER API, or via a nice Perl or XSL:T script, Converter allows some of the more common modifications directly, as well as some interesting ones via a filter.

If a filter is passed to a Converter, it must contain a FILTER region, marked with #BEGIN_FILTER and #END FILTER. The filter file format is specified in the ViPER-PE manual. While most of the filters, like size, contains and ==, determine if a descriptor is to be output, a few filters, like resize and cropTo, are designed to modify the attribute values. For example, to modify a ground truth file that has bbox attributes that go out side the frame, pass a cropTo ("0", "0", "640", "480") to the appropriate attribute of the descriptor in the FILTER, where the frame size is 640x480.

Adding Data Types to ViPER-GT

To add a data type to ViPER-GT, you must add: the necessary extensions to the parser and serializer to XML, the Java JTable cell renderer and editor, any extended support for editing the attribute configuration in the schema editor, and, finally, any canvas views and controllers or timeline renderers if requested.

The first extension is simply reading in the data. This involves adding a new data wrapper. You can see some examples in the viper.api.datatypes package. For a simple example, see the viper.api.datatypes.Bvalue , while the viper.api.datatypes.Polygon is perhaps the most complex example. The attribute wrapper must implement AttrValueParser to support serialization and deserialization, and DefaultedAttrValueWrapper to support taking default values. (If you wish to specify parameters for the data in the schema section, implement ExtendedAttrValueParser instead of AttrValueParser .)

An AttrValueWrapper is a utility object that enforces typing on the attributes. It can also convert from an internal format (say, an integer lookup into a table, in the case of an enumeration) to an external format (such as a String ). This allows space savings and possibly allows the use of == instead of .equals() . The interface we are implementing, AttrValueParser , extends this to support serialization to an XML DOM object. Note that the de/serialization functions use the encoded format of the data.

Perhaps it is a good idea to mention something about how best to formulate the XML syntax for encoding your data. I would recommend developing a schema data type for your attribute, as I have done for each of the currently included data types in the the ViPER Data Type schema . Again, it is simple enough to stuff everything into a 'value' attribute of an XML element, but you may wish to have multiple XML attributes (e.g. the bounding boxes' x , y , width and height ), or you may wish to have nested elements (like the polygons' data:point elements). To add the data type to the list of allowed data types, you can either modify the ViperDataFactoryImpl class, or you can add your classes using the ViperDataFactoryImpl.addType method. You will have to do this before parsing. If you wish to modify ViPER-GT to support your data type, you can add this by modfying the 'datatypes' bean declaration in the gt-config.n3 file. [TODO:: explain how; possibly requires modifying apploader]

After you implement the api, you can run the Java command line tool viper.api.impl.ViperParser.FileTranscriberTest , which parses a file and serializes it again. This should allow you to check for errors both in your format, assuming your code is written correctly, and your code, assuming your XML is valid.

[TODO:: add information about adding jtable cell renderer and editor]

[TODO:: add information about adding new data type to video frame]

For information on adding your own wizards and views, see the Extending GT section below.

Adding Data Types to ViPER-PE

To add an attribute data type to ViPER-PE, a programmer must subclass viper.descriptors.attributes.Attribute (or one of its subclasses) and implement a set of methods. The class name must have the form: viper.descriptors.attributes.Attribute_*datatype*, where datatype is replaced with the name of the attribute type. This class acts as a proxy for the actual data type, which must implement the AttributeValue interface, and may implement the ``ExtendedAttributeValue` interface`_ if the attribute takes additional information in its configuration section. Most of the mechanics of the data type are performed in the root Attribute class; the proxy subclass must only follow a few rules.

These rules include implementing three constructors, all three of which must first pass their parameters to the super constructor, and then do what is necessary to call setArchetype. The three constructors are the default constructor, a constructor that takes a single boolean indicating the attribute is dynamic, and a constructor that takes an Attribute, from which to create a duplicate. The archetype is an instance of the AttributeValue class that the new data type will use, and the root class uses its factory methods to create additional instances without having to know about its type. Note that instances of AttributeValue should be immutable. The Attribute class must also implement a clone() method, but all other methods are optional.

If your data type takes extended information, such as the list of possible values for an lvalue, its archetype must implement the ExtendedAttributeValue interface, which gives methods for getting the extended config string for output. Since AttributeValues must be immutable, there is no method to modify the extended attributes. Instead, the Attribute proxy class must implement setArchetype (String) and setArchetype (Element) (or only one or the other if you donÂ't wish to support both data formats).

If you wish to make your data type composable, for use with the MULTIPLE OBJECT_EVALUATION, have the data type implement the ``Composable` interface`_.

An Example: Labelled Text Zones

One common form of ground truth is labelled text zones, for example, in a document. This form of ground truth is often, these days, generated directly from source. However, we are finding it again useful to test camera based OCR, new script identification algorithms, and any system that works on older data.

We will use a simple example schema, with a single OBJECT descriptor type called ZONE, with three attributes: an enumeration indicating the zone type, a bounding box indicating its location, and a relation indicating its place in a directed graph of other ZONE descriptors.