`hydrate-protobuf` (Transform Node)

The Hydrate Protobuf node reads binary or JSON payloads from a HEAT capture bundle (e.g. from an upstream ingest or node that emits DataPacketBundle-style data), extracts a specific protobuf type from the bundle, and writes the hydrated payloads as JSON, CSV, or raw binary.

Use this node when you need to turn opaque protobuf blobs into a structured format (JSON/CSV) for downstream analytics, dashboards, or export. The node supports multiple inputs and can aggregate simulation frames into a single time-ordered stream when configured.

Configuration Schema

Property	Type	Required	Description
`targetProtoType`	`string`	Yes	Fully-qualified protobuf type inside the bundle. Supported values depend on the runner implementation (e.g. `GazeComponent`, `FolderUploadData`, `HEAT.OpenDISModule.PDU`).
`outputFormat`	`"json"` \| `"csv"` \| `"bin"`	Yes	Encoding for the output. `json` and `csv` produce human-readable or tabular data; `bin` writes the raw serialized protobuf bytes. Defaults to `json`.
`lineDelimitedJson`	`boolean`	No	Additive. When `true`, valid with `targetProtoType` `SimulationFrame` or `HEAT.OpenDISModule.PDU`, `outputFormat` `json`, and `includeEvents` false. Emits newline-delimited JSON (JSON Lines): each line is one JSON object with the same shape as one element of the legacy wrapper array (`simulation_frame` or `open_dis_pdu`), not the wrapped document. Default `false` preserves the single JSON object with that array. Downstream `json-flatten` should use `jsonInputFormat: "ndjson"` and usually empty `root`. Memory: when `lineDelimitedJson` is `true` and `incrementalHydration` is `false`, the runner streams decode (SimulationFrame also k-way merges) so full replay stays within cgroup memory. For SimulationFrame only, set env `HEAT_RUNTIME_HYDRATE_BUFFER_SIMFRAME=1` to restore the old in-memory path.
`aggregateSimulationFrames`	`boolean`	No	When `targetProtoType` is a simulation frame type, combine all frames into a single time-ordered stream. Defaults to `true`.
`emitFrameIndex`	`boolean`	No	When aggregating frames, include a synthetic frame index column/field in CSV/JSON. Defaults to `true`.
`enableUploadDeduplication`	`boolean`	No	Compute an MD5 hash of the output and skip uploading if the same hash already exists for this node instance. Defaults to `false`.
`incrementalHydration`	`boolean`	No	Opt-in. When `true`, only decode parent outputs newer than the watermark stored on this node’s prior output, and append to a cumulative artefact using streaming I/O. For `outputFormat: json`, requires `lineDelimitedJson: true` (NDJSON). For `HEAT.OpenDISModule.PDU` with `outputFormat: csv`, appends CSV rows and stores `csvHeaders` on the output config (no `lineDelimitedJson`). Default `false` preserves full replay (legacy behaviour).
`csvInclude`	`string[]`	No	Additive. OpenDIS `outputFormat: csv` only. Whitelist of dotted column prefixes (same semantics as `json-flatten` `include`). A flattened key is kept when it equals an entry or starts with `entry + "."`. Omitted or empty keeps all columns. When set, also stored on the output config next to `csvHeaders`.

Incremental hydration (opt-in)

When incrementalHydration is true (default false):

The node processes only new parent NodeOutput records since the last successful run (watermark: incrementalLastParentOutputId on the node’s output configuration).
Prior artefact bytes are stream-copied from object storage into a temp file. New rows are written as CSV segments during decode; header growth opens a new segment and at most one merge runs at the end of the run (no full NDJSON value spill). Status heartbeats keep long first-pass hydrates from being reaped by the Core API stale-task timeout. Identical cumulative md5 skips re-upload so latest-only children (for example tabular-query) are not rescheduled for no content change.
Each run still produces one cumulative artefact. Downstream nodes should keep inputMode: "latest" (do not switch to all).
NDJSON path: targetProtoType SimulationFrame or HEAT.OpenDISModule.PDU, outputFormat: json, and lineDelimitedJson: true.
CSV path (OpenDIS): targetProtoType HEAT.OpenDISModule.PDU, outputFormat: csv. Rows use dotted keys compatible with json-flatten DIS CSVs (for example pdu.pduType). Header list is persisted as csvHeaders on the output config. New columns discovered in a run expand the header list with at most one rewrite of the prior cumulative CSV for that run (new rows are spilled during decode, then materialised). Optional csvInclude can narrow columns when a template intentionally wants a subset; omit it to keep the full flattened DIS surface for SQL. Prefer this path when the next step is tabular-query (skip json-flatten).
Downstream contract: the published artefact is always cumulative (prior rows + new). tabular-query only reads the parent latest output, which is correct for this path. Do not combine all historical hydrate outputs (that would double-count). If the prior cumulative CSV cannot be streamed on a later run, the node fails rather than emitting a delta-only file.
Output config also records cumulativeArtefact: true, csvRowCount, and csvRowsAdded for operator visibility.

Supported protobuf types

The runner implements decoders for a fixed set of types. Typical values include:

SimulationFrame , Per-component rows from simulation frames; JSON output is typically an object with a simulation_frame array, or NDJSON (lineDelimitedJson) with one row object per line.
GazeComponent , Gaze and biometric sensor data; output is JSON with a gaze_component array.
FolderUploadData , Uploaded file/folder content; time-series CSV content is merged into a single artefact.
HEAT.OpenDISModule.PDU , OpenDIS PDU records; default JSON is {"open_dis_pdu":[...]}, or NDJSON (lineDelimitedJson) with one flattened PDU record per line, or incremental CSV (outputFormat: csv + incrementalHydration) for a cumulative tabular artefact. Optional excludePduTypes (for example [26] for Signal) drops PDU types during decode. Optional csvInclude narrows CSV columns.

If you specify a targetProtoType not implemented by the runner, the node fails with an error. Check the runner’s supported types when authoring the configuration.

Example configuration


{
  "targetProtoType": "HEAT.OpenDISModule.PDU",
  "outputFormat": "json",
  "aggregateSimulationFrames": true,
  "emitFrameIndex": true,
  "enableUploadDeduplication": false
}

Inputs and outputs

Inputs: The node accepts multiple inputs (e.g. from a single upstream node that emits a list of payloads, or from several nodes). All payloads are collected and decoded according to targetProtoType.
Output: One or more artefacts are produced. For outputFormat json or csv, the content is UTF-8 encoded. For bin, the artefact is the raw serialized protobuf bytes.

Integration in a session template

Place a hydrate-protobuf node after a node or ingest that produces HEAT capture bundles (or raw protobuf payloads the runner can decode).
Set targetProtoType to the type you want to extract and outputFormat to the desired encoding.
Connect downstream nodes (e.g. tabular-query, json-template, or dashboard nodes) to consume the hydrated output.

Core Utils Runner
Node Templates
Tabular Remap or tabular-query (for further processing of CSV output)

hydrate-protobuf (Transform Node)