Create a Custom Datatype

The data transferred between DV modules is implemented as FlatBuffers. This tutorial is to show how to create a new datatype based on FlatBuffers which can then be used to transmit data between different modules.

Prerequisites

First, the DV-runtime needs to be installed on your machine. Additionally, the following tools need to be available:

cmake
clang-format
make
bash

How-To

Getting the Schema Compiler

The precompiled binaries for the schema compiler flatc are located in the dv-runtime repository. The precompiled binaries are located in dv-runtime/flatbuffers. For more information on the schema compiler flatc, check out the official documentation: https://google.github.io/flatbuffers/flatbuffers_guide_using_schema_compiler.html

Writing the Schema

The full syntax for the schema file is documented in the following guides:

An example for a schema file is shown below:

native_include "dv-sdk/data/cvector.hpp";

file_identifier "PLOT";

struct Vector3D {
	x: int32;
	y: int32;
	z: int32;
}

table PlotPoint {
	/// Timestamp (µs).
	timestamp: int64;
	/// Plot X coordinate.
	coord: Vector3D (native_inline);
}

table PlotPacket {
	elements: [PlotPoint] (native_inline);
}

root_type PlotPacket;

This schema creates a FlatBuffers vector which can be used as an input or output throughout the DV toolchain using the dv-sdk. The vector is of type PlotPacket and contains elements of type PlotPoint. Each PlotPoint contains a timestamp and a Vector3D.

Remarks Regarding the Schema

In order for the schema to be compatible with the DV toolchain it is necessary for it to contain a variable called timestamp of type int64.
The attribute native_inline is used whenever the class created by flatc should contain an object of the struct for which the attribute is dedicated. If the attribute is not present, the field will be implemented as a pointer.
tables may be extended in future versions of the FlatBuffer that is created, structs may not.
The file_identifier needs to be a string of four characters

Compiling the Schema

In order to compile the schema for use with the dv-sdk the following command should be used:

<path_to_flatc> --cpp \
                --scoped-enums \
                --gen-object-api \
                --gen-compare \
                --gen-name-strings \
                --cpp-ptr-type "std::unique_ptr" \
                --cpp-str-type "dv::cstring" \
                --cpp-str-flex-ctor \
                --cpp-vec-type "dv::cvector" \
                --reflect-types \
                --reflect-names <schema_file>.fbs

mv `basename <schema_file>`_generated.h <schema_file>_base.hpp
clang-format -i <schema_file>_base.hpp

For simplicity, this has been implemented in a bash script in the dv-runtime repository: flatbuffers/dv-sdk-flatc.sh (see https://gitlab.com/inivation/dv/dv-runtime/-/tree/master/flatbuffers)

You can also just clone the dv-runtime repository instead and execute the file directly.

Using the FlatBuffer Object

Now that the <schema_file>_base.hpp object is created, it can be used with the dv-sdk.

The following examples show how the FlatBuffer object can be used in a Module.

Registering your Custom Type with the Module

Add the following static function to your module class.

static void initTypes(std::vector<dv::Types::Type> &types) {
	types.push_back(dv::Types::makeTypeDefinition<PlotPacket, PlotPoint>("A point to be plotted"));
}

Adding the FlatBuffer Object as an Output of a Module

static void initOutputs(dv::OutputDefinitionList &out) {
  out.addOutput("stats", PlotPacket::TableType::identifier);
}

The value of PlotPacket::TableType::identifier is the four character file_identifier (in this case "PLOT").

Adding an Object to the FlatBuffer Output Vector

auto stats  = outputs.getVectorOutput<PlotPacket, PlotPoint>("stats").data();

Vector3D vec(1, 2, 3);
PlotPoint point(<timestamp>, vec);

stats.commit();

Adding the FlatBuffer Object as an Input of a Module

static void initInputs(dv::InputDefinitionList &in) {
  in.addEventInput("stats", PlotPacket::TableType::identifier);
}

The value of PlotPacket::TableType::identifier is the four character file_identifier (in this case "PLOT").

Getting an Object from the FlatBuffer Input Vector

void run() {
  auto stats  = outputs.getVectorInput<PlotPacket, PlotPoint>("stats").data();

  for(const auto &point: stats) {
    // do whatever
  }
}