ADTF3 Guides - ADTF DAT Tool

ADTF DAT Tool

Since ADTF version 3.3.0 there is a extendable command line tool - based on ADTF File Library - to handle and access ADTFDAT Files for read, write and process containing streaming data using processors and readers, which can be implemented by the use of adtfdat_processing library. The ADTF File Library will be redeployed within ADTF, the SDK, reader/processors and examples can be found under $(ADTF_DIR)/pkg/adtf_file, the ADTF DAT Tool will be next to the other applications under $(ADTF_DIR)/bin.

ADTFDAT files contain one or more streams representing recorded data. The ADTF DAT Tool operates on these streams in one of two ways:

Export Mode

In this mode the ADTF DAT Tool makes use of processors to operate on streams inside the ADTFDAT file. processors are written by software developers to do versatile data manipulation and extraction. The file ending of a processor is *.adtffileplugin.

Currently the ADTF delivery provides one such processor to export the samples of a stream into a CSV file (see ADTF File Library). Another processor can be found in the programming examples of this guide which is capable of extracting images from a video stream. Addional content can be found within the toolboxes (ASC/MDF within Device and Calibration Toolbox) and standalone solutions from digitalwerk (see artifactory and store) and others.

To extract data from streams of an ADTF DAT file use the --export argument. Select the streams you want to extract by specifying the --stream argument. You can specify a processor for each stream with the --processorid argument. If you do not specify one explicitly, the first one that supports the stream is used. Properties of processors can be specified with the --property argument. The destination filename is specified with the --output argument. Here is an example that exports two streams:

          
adtf_dattool --plugin csv_processor.adtffileplugin --export test.dat --stream in1 --processorid csv --output test_in1.csv --stream in2 --output test_in2.csv

Apply the parameter --liststreams to query all information about a given input (DAT file, or any other supported input). To load additional plugins use the --plugin argument as often as you like.

Create Mode

In this mode you can select arbitrary streams (even from multiple DAT files) to create a new DAT file. To do so use the --create argument. The --input argument is used to specify an input file. The --readerid argument can be used to pass in the reader that should be used to read the file. If none is given, the first one that supports the file is used. Set the --start and --end arguments to select the range of the input that should be imported into the new DAT file. To shift the timstamps of all imported stream items apply the --offset parameter.
To select streams from an input, use the --stream argument. If you do not select one or more streams explicitly, all streams will be added. Streams can be renamed with the --name argument.
Use the --serializerid argument to choose the serializer of your liking. If not specified, sample_copy_serialization.serialization.adtf.cid will be used.
Here is an example that creates a new ADTF DAT file from two inputs:

        
adtf_dattool --create new.dat --input input1.dat --readerid adtfdat --stream in1 --serializerid special.serialization.adtf.cid --input input2.dat

Best Practice

This chapter should show you a workflow how to start from scratch to use the ADTF DAT Tool, extend the functionality using adtffileplugins and how to export containing streams from a recording. The separated steps will teach you about architecture and how to components and functionality are connected. For a better overview, we strip the paths from the call and arguments.

Get information about the tooling

First of all, the command line help should always be the first address for information, no matter as getting started or for a deeper dive later:

          
adtf_dattool --help

Get information from the adtfdat file

As next step you should have a look at the recorded file, in this example the example_file.adtfdat within $(ADTF_DIR)/src/examples/datfiles. The --liststreams option will list the stored data sources, in this case, one video and one ddl described stream.

          
adtf_dattool --liststreams example_file.adtfdat
    adtfdat:
        VIDEO:
            type: adtf/image
            processors:
            time range (ns): [405356000, 14805306000]
            items: 874
        NESTED_STRUCT:
            type: adtf2/legacy
            processors:
            time range (ns): [0, 14805306000]
            items: 595

Get information from the adtffileplugin

With the knowledge about the containing streams, it is time to find a suitable processor to export some data. The --inspect option can be used to look inside an adtffileplugin, in this case the csv_processor.adtffileplugin with the ADTF File Library besides the ADTF DAT Tool itself.

          
adtf_dattool --inspect csv_processor.adtffileplugin
{
  "processors": [
    {
      "id": "csv",
      "properties": [
        {
          "name": "decimal_places",
          "type": "uint32",
          "value": "4"
        },
        {
          "name": "separator",
          "type": "string",
          "value": ";"
        },
        {
          "name": "timestamp_base_ns",
          "type": "uint64",
          "value": "1"
        },
        {
          "name": "timestamp_decimals",
          "type": "uint32",
          "value": "0"
        }
      ]
    }
  ],
  "readers": [
  ],
  "sample_serializers": [
  ],
  "sample_deserializers": [
  ],
  "type_serializers": [
  ],
  "type_deserializers": [
  ]
}

Extend the functionality

With this information, the calls can be extend by using --plugin which provides the csv processor to export the stream NESTED_STRUCT compared to the --liststreams call without that adtffileplugin.

          
adtf_dattool --liststreams example_file.adtfdat --plugin csv_processor.adtffileplugin
    adtfdat:
        VIDEO:
            type: adtf/image
            processors:
            time range (ns): [405356000, 14805306000]
            items: 874
        NESTED_STRUCT:
            type: adtf2/legacy
            processors: csv
            time range (ns): [0, 14805306000]
            items: 595

Export the data

Now all required information for exporting are available:

the tool call from the help

wa valid adtfdat file

the name of streams

an adtffileplugin containing logic to export data

the suitable processorid for csv export

information about optional properties

          
adtf_dattool --plugin csv_processor.adtffileplugin --export example_file.adtfdat --stream NESTED_STRUCT --processorid csv --output exported_data_as.csv

Use cases for using properties for this processor could be adapting the decimal places:

          
adtf_dattool --plugin csv_processor.adtffileplugin --export example_file.adtfdat --stream NESTED_STRUCT --processorid csv --output exported_data_as.csv --property decimal_places=1

or exporting timestamps in microseconds:

          
adtf_dattool --plugin csv_processor.adtffileplugin --export example_file.adtfdat --stream NESTED_STRUCT --processorid csv --output exported_data_as.csv --property timestamp_base_ns=1000

Handle substreams

Processing substreams is almost the same workflow, with tiny extensions. The --liststreams command will show you the content as already learned:

          
adtf_dattool --liststreams example_file_substreams.adtfdat
    adtfdat:
        substreams:
            type: adtf/substreams
            processors:
            time range (ns): [66746000, 14805306000]
            items: 1467

The important identifier in this case is the type adtf/substreams for the stream substreams. So you have to dive deeper using --listsubstreams:

          
adtf_dattool.exe --liststreams example_file_substreams.adtfdat --listsubstreams substreams
    adtfdat:
        substreams:
            NESTED_STRUCT:
                type: adtf2/legacy
                processors:
            VIDEO:
                type: adtf/image
                processors:

You will receive the same required information compared to the common sample streams use case. Now you can also extend the functionality using a processor and export csv in the same manner (we skip the step with a relook and inspect because we already know the information from our previous steps). The only difference during the --export call is to add the (containing) --substream after the specified --stream:

          
adtf_dattool --plugin csv_processor.adtffileplugin --export example_file_substreams.adtfdat --stream substreams --substream NESTED_STRUCT --processorid csv --output exported_data_as.csv

Side note

Now you have a good start and also common workflow for the export use case. Of course you can use this procedure with less adaption for create and modify option as well. Please keep also in mind that you can load several adtffileplugins at once, each with an own --plugin call to load readers and processors as a kind of file conversion or several readers and processors for exporting more than one stream with processors from different adtffileplugins (that's why it is important to address them by using --processorid). For more help, please always refer to --help which provides you the information as follows:

Full Command Line Help

The following lists all options the tool supports. Just issue the following command: adtf_dattool.exe --help

        
usage:
  adtf_dattool  options

where options are:
  -?, -h, --help                         display usage information
  --progress                             Show progress.
  --skipstreamtypesandtriggers           Do not process stream types and
                                         triggers.
  --plugin <adtffileplugin>              Load an additional plugin.
  --inspect <adtffileplugin>             Inspect a plugin and print its
                                         contained items.
  --inspect-builtins                     Inspect all built-in classes.
  --json                                 Output information in json format.
                                         Raw timestamps within the json
                                         output are always output as seconds.
  --liststreams <file name>              List all available information about
                                         the given file.
  --listsubstreams <stream name>         List all substreams available in the
                                         given stream.
  --export <file name>                   Export streams from the given file.
  --create <file name>                   Create a new adtfdat file.
  --modify <file name>                   Modify an existing adtfdat file.
  --dump <file name>                     Dump information about an adtfdat
                                         file. If no other options (--header,
                                         --streams, --stream, --extensions, -
                                         -extension) are additionally
                                         specified, an entire overview will
                                         created.
  --header                               Show header information for a
                                         adtfdat file in combination with --
                                         dump.
  --streams                              Show all stream information for a
                                         adtfdat file in combination with --
                                         dump.
  --extensions                           Show all extension information for a
                                         adtfdat file in combination with --
                                         dump.
  --fileversion <adtf2|adtf3|adtf3ns>    File Version of the created file.
                                         Creating ADTF 2 files is completely
                                         experimental!
  --input <source>                       Specifies an input for the new
                                         adtfdat file or extension. In case
                                         of extensions data will be read from
                                         stdin if this is not specified.
  --readerid <reader id>                 The id of the reader implementation
                                         that should be used to open the last
                                         list/input/export source.
  --referencedfiles                      Process referenced files (i.e.
                                         splits) from the last input/export
                                         source as well.
  --ignoreopenerrors                     Ignore errors when referenced files
                                         cannot be opened.
  --start <timestamp>                    Process only chunks of the last
                                         input source with timestamps larger
                                         than this. @See List of supported
                                         timestamp formats.
  --end <timestamp>                      Process only chunks of the last
                                         input source with timestamps smaller
                                         than this. @See List of supported
                                         timestamp formats.
  --offset <timestamp>                   This offset is added to all chunk
                                         timestamps of the last input source.
                                         @See List of supported timestamp
                                         formats.
  --stream <stream name>                 Select a stream for dump, export,
                                         modify existing or create new
                                         adtfdat file.
  --stream-type <stream name>            Dump stream type of the given
                                         stream.
  --ddl <stream name>                    Dump ddl from stream type of the
                                         given stream.
  --substream <substream name>           Select a substream for export. Keep
                                         in mind that if you want to export
                                         multiple substreams from the same
                                         stream, you need to specify --stream
                                         for each substream. If the substream
                                         is not available within the initial
                                         stream type you always need to
                                         specify a processor id with --
                                         processorid.
  --name <stream name>                   Sets the name for the last specified
                                         stream that is used in the newly
                                         created adtfdat file.
  --processorid <processor id>           The id of the processor
                                         implementation used for the last
                                         specified stream or substream.
  --property <name=value>                Sets a property of the last
                                         specified export job, stream or
                                         input.
  --extension <extension name>           Select an extension for dump,
                                         export, modify existing or create a
                                         new adtfdat file.
  --output <file name>                   Sets the output file name for the
                                         last specified stream or extension.
                                         If not specified, the processor is
                                         free to choose one and in case of
                                         extensions data will be written to
                                         stdout
  --serializerid <serializer id>         The id of the serializer
                                         implementation used for the last
                                         specified stream.
  --userid <user id>                     Sets the user id of the last
                                         extension to be updated.
  --typeid <type id>                     Sets the type id of the last
                                         extension to be updated.
  --versionid <version id>               Sets the version id of the last
                                         extension to be updated.
  --rename <new name>                    Renames the given stream.
  --repair <file name>                   Repair the given corrupted file.
                                         Requires --reference and --output as
                                         well.
  --reference <file name>                The reference file for the repair
                                         operation.


-----------------
  ADTF DAT Tool
-----------------

With the help of this tool you can extract data from adtfdat files, create new adtfdat files
from various inputs or add file extensions in combination with the adtf_datdump tool
to an existing adtfdat file.

Use --liststreams to query all information about a given input (adtfdat file, or any other
supported input).

To load additional adtffileplugins use the --plugin argument as often as you like.

-----------
  EXPORT:
-----------
To extract data from streams or file extensions of an adtfdat file use the --export argument.
Select the streams you want to extract by using the --stream argument. You can specify a processor
for each stream with the --processorid argument. If you do not specify one explicitly, the first one
that supports the stream is used. Properties of processors can be specified with the --property
argument.

Use the --extension argument to specify the required file extension. The destination
filename is specified with the --output argument. Without --output argument the file extension data
will be written to stdout.

To process a set of files in one go specify '--readerid multiple_files' right after your
--export or --input argument and specify the set of files as a ';' seperated list.

To process referenced files transparently specify --referencedfiles right after your
source adtfdat file.

Examples:
---------
Here is an example that exports two streams:
adtf_dattool --plugin csv_exporter.adtffileplugin --export test.dat --stream in1
--processorid csv --output test_in1.csv --stream in2 --output test_in2.csv

And one that exports extension data:
adtf_dattool --export test.dat --extension adtf_version --output adtf_version.txt

Process referenced files as well:
adtf_dattool --plugin csv_exporter.adtffileplugin --export first_split_001.dat
--referencedfiles --stream in1 --processorid csv --output in1.csv

Export only part of input limited by global 'start' and or 'end' timestamps:
adtf_dattool --plugin csv_exporter.adtffileplugin --export example_file.adtfdat
--stream NESTED_STRUCT --processorid csv --output cutted.csv --start 7.7s --end 10s

-------------
  CREATION:
-------------
To create a new adtfdat file use the --create argument. The --input argument is used to specify an
input file. The --readerid argument can be used to specify the reader that should be used to read
the file. If none is specified, the first one that supports the file is used.

Use the --start and --end arguments to select the range of the input that should be
imported into the adtfdat file. Use the --offset parameter to shift the timstamps of all imported
stream items.

To select streams from an input, use the --stream argument. If you do not select one or more
streams neither a extension, all streams will be added. Streams can be renamed with the --name argument.

Use the --serializerid argument to choose the serializer of your liking. If not specified,
sample_copy_serialization_ns.serialization.adtf.cid will be used.

To select file extensions from an input, use the --extension argument. If you do not select one or
more file extensions explicitly, no extension will be added or updated from the source file.

Examples:
---------
Here is an example that creates a new adtfdat file from two inputs:
adtf_dattool --create new.adtfdat --input input1.dat --readerid adtfdat --stream in1 --serializerid
special.serialization.adtf.cid --input input2.dat

And one that copies a stream and two extensions from a source file
adtf_dattool --create new.adtfdat --input input1.dat --stream in1 --extension attached_files
--extension attached_files_configuration

------------------------
  MODIFYING:
------------------------
To modify an existing adtfdat file use the --modify argument.
Currently there is only support for adding and updating extensions.

Examples:
---------
An example to put a files content into an extension:
adtf_dattool --modify existing.adtfdat --extension my_extension --input input_file.txt

Mind that if you do not specify an --input for an extension its data will be read from stdin:
tar cz /myfolder | adtf_dattool --modify existing.adtfdat --extension attached_files

or use:
adtf_dattool --export source.adtfdat --extension attached_files | adtf_dattool --modify destination.adtfdat --extension attached_files

to copy a explicit file extension of an existing adtfdat file to another.

To rename a stream use:
adtf_dattool --modify existing.adtfdat --stream old_name --rename new_name

------------------------
  DUMP:
------------------------
To show some information about an adtfdat file itself and its entire content you can use '--dump <file name>'. This works also in combination with '--streams' or '--header'
to print these specifiic stream or extension information. Furthermore, you can dump also the content for an explicit '--stream <stream name>' or '--extension <extension name>'.

------------------------
  INSPECTION:
------------------------
To inspect an adtffileplugin use the --inspect argument. This will list all classes and their configuration
properties contained within the given plugin file. To list all built-in classes use '--inspect-builtins'.

------------------------
  TIMESTAMP FORMATS:
------------------------
Supported are the following ISO-8601 formats:
    e.g. 2024-09-26T23:59:59Z   -> Universal time UTC-0 (GMT). (preferred)
    e.g. 2024-09-26T23:59:59    -> Local time.
    e.g. 2024-09-26T23:59:59.123456789Z

    With timezone postfix 'Z' (default) means GMT time (UTC-0). Universal Coordinated Time.
    No timezone postfix 'Z' means your local timezone.

Supported is the number+unit format:
    with several unit postfixes = {h|hh|m|mm|min|s|ss|sec|ms|us|ns}
    e.g. 14.125h
    e.g. 8123ms
    e.g. 3.141s

Unsupported formats:
    e.g. 9.12      (must have unit)
    e.g. T23:59:59 (no time only ISO timestamps)
    e.g. 1h 30ms   (no combined time fragments)

Where to go next?

ADTF itself is a large bundle of SDKs and tooling. It has its limitations but is able to work in a distributed setup. We think it is time to step across this border and give you an introduction in the ADTF FEP Integration.