hydrate-protobuf-v2 (Transform Node)

The Hydrate Protobuf v2 node decodes HEAT capture data shaped as DataPacket / DataPacketBundle (see HEAT.Proto.Capture in the protolib), walks every nested packet, and extracts a single kind of simulation payload from each SimulationFrame: the full frame, each EntityState, each EntityDataEvent (from frame.events where the EntityEvent carries data_event), or each TrackingEvent (from tracking_event).

Use this node when you need capture-aware extraction with stable envelope fields (parent input key, packet timestamps, origin_uuid, simulation tick/time) across all upstream parents. Optional projection filters prune components and identity so domain trees avoid multi‑hundred‑MB overhydration. The original hydrate-protobuf node remains for Gaze, FolderUploadData, OpenDIS PDU, and its legacy SimulationFrame component flattening.

Capture hierarchy

Configuration schema

When no projection filters are set, behavior matches the previous full-tree extraction.

Incremental hydration (opt-in)

Incremental hydration still re-uploads the whole cumulative NDJSON on every run where the bytes changed. For live Capture lanes use hydrate-protobuf-journal with extractionTarget and journalFormat: "ndjson", which publishes only the new records as an append-only part.

Same contract as hydrate-protobuf: cumulative NDJSON per run, watermark on output config, downstream inputMode: latest. Supported for EntityDataEvent and TrackingEvent extraction.

Live anti-baggage: when the newly built artefact has the same md5 as the node’s current output, the processor completes without uploading. That matches OpenDIS CSV incremental behaviour and avoids creating a new NodeOutput (and child fan-out) on every agnostic-input tick when nothing in this lane changed. The watermark stays on the prior output until content actually changes.

For button_press streams, combine buttonPressEdgesOnly with incrementalHydration so the cumulative artefact stays small and only grows when real edges appear.

Records-JSON variant (cumulativeRecordsJson: true): same watermark and md5-skip contract, but the artefact stays a records object. With buttonPressEdgesOnly, the edge filter state (last held sample per origin, entity and button) is persisted as buttonPressEdgeState on the output configuration and seeded on the next run, so a hold that straddles two triggers is not re-emitted as a new press. On an md5 skip the watermark and edge state are patched onto the prior output. Samples older than the seeded state (retroactive uploads) fall back to state-change detection only; a full replay remains the order-exact path.

Output shape (JSON)

The artefact is a JSON object:

{
  "extractionTarget": "EntityState",
  "records": [
    {
      "parent_input_key": "upstream-node-name",
      "payload_index": 0,
      "inner_packet_index": 2,
      "bundle_timestamp_unix_ms": null,
      "packet_timestamp_unix_ms": 1710000000000,
      "packet_timestamp_iso_utc": "2024-03-09T12:00:00.000Z",
      "origin_uuid": "…",
      "simulation_tick": 42,
      "simulation_time_seconds": 1.5,
      "capture_time_timestamp_ms": null,
      "capture_time_iso_utc": null,
      "capture_time_timezone_offset_minutes": 0,
      "extraction_target": "EntityState",
      "entity_state_index": 0,
      "entity_state": { }
    }
  ]
}

For EntityDataEvent / TrackingEvent, each record follows the decoder event dictionary layout (event_category, event_type, event_data, …) plus the same envelope fields merged in.

Example configurations

Full EntityState (legacy behavior):

{
  "extractionTarget": "EntityState",
  "outputFormat": "json",
  "sortBy": "simulation_time_seconds",
  "dataSourceName": "my-store"
}

Domain tree (world position only, named entities):

{
  "extractionTarget": "EntityState",
  "outputFormat": "json",
  "includeComponentTypes": ["world_position", "dynamics"],
  "entityNameRegex": "^(Blue|Red)-.*",
  "dropEmptyEntities": true
}

Tracking timeline for one entity:

{
  "extractionTarget": "TrackingEvent",
  "outputFormat": "json",
  "lineDelimitedJson": true,
  "incrementalHydration": true,
  "filterEntityId": 42,
  "includeTrackingEventTypes": ["TRACKING_EVENT_TYPE_FIRED", "TRACKING_EVENT_TYPE_HIT"]
}

Inputs and outputs

• Inputs: All parents are merged. Each parent contributes a list of binary DataPacket payloads (same wire format as v1).
• Output: One JSON or CSV artefact. Unlike v1, every parent key in fetch_all_inputs is processed, not only the first.

Inline media content

Some capture emitters put a media stream on a string field instead of emitting it on its own channel. A radio feed can arrive on CommsEvent.content as:

audio/pcm;src=mic;rate=48000;bits=32;ch=2;enc=base64,<megabytes of base64>

Every artefact, merge and dashboard script downstream of the hydrate node then carries those bytes, usually so that the single consumer which reads the field can discard them. With incrementalHydration and cumulativeRecordsJson the payload is also held in memory for the whole run.

inlineMediaContentPrefixes truncates such a value at the first comma, keeping the header and dropping the payload:

{ "inlineMediaContentPrefixes": ["audio/"] }

The header is kept rather than blanking the field so that the record stays self-describing: a reader can still see the declared sample rate and bit depth, and a MIME-prefix test still matches, at a few tens of bytes instead of the payload. An empty string would also make “this was media, handled elsewhere” look identical to “this event genuinely had no content”.

Defaults, and why they should stay

Unset or empty is a no-op. Any node that says nothing about inline media behaves exactly as it did before this option existed.

Do not make truncation the default. It will look like an obvious improvement later, because nobody wants megabytes of base64 in a JSON artefact. But a consumer that legitimately wants the inline payload out of this node would then get an empty header with no error and no warning, and hydrate-protobuf-v2 is a long-lived contract whose changes are additive by default.

It is lossy here, not lost

The raw capture blob keeps the full payload, so nothing is destroyed. A consumer that needs the bytes reads the blob directly, as hydrate-protobuf does when decoding CommsEvent audio (see hydrate-protobuf), or the node is reprocessed with no prefixes configured.

The truncation is applied on this node’s EntityDataEvent extraction only, never in the shared protobuf decoder, precisely because hydrate-protobuf reads the same field to recover that audio.

Related

• Core Utils Runner
• Hydrate Protobuf (v1)
• system-protobuf-extract (platform subset with the same projection knobs)
• Node Templates