`hydrate-protobuf-v2` (Transform Node)

The Hydrate Protobuf v2 node decodes HEAT capture data shaped as DataPacket / DataPacketBundle (see HEAT.Proto.Capture in the protolib), walks every nested packet, and extracts a single kind of simulation payload from each SimulationFrame: the full frame, each EntityState, each EntityDataEvent (from frame.events where the EntityEvent carries data_event), or each TrackingEvent (from tracking_event).

Use this node when you need capture-aware extraction with stable envelope fields (parent input key, packet timestamps, origin_uuid, simulation tick/time) across all upstream parents. Optional projection filters prune components and identity so domain trees avoid multi‑hundred‑MB overhydration. The original hydrate-protobuf node remains for Gaze, FolderUploadData, OpenDIS PDU, and its legacy SimulationFrame component flattening.

Capture hierarchy

Configuration schema

Property	Type	Required	Description
`extractionTarget`	`string`	Yes	One of `SimulationFrame`, `EntityState`, `EntityDataEvent`, `TrackingEvent`.
`outputFormat`	`"json"` \| `"csv"`	Yes	`json`: UTF-8 JSON object with a top-level `records` array. `csv`: scalar columns plus `record_json` (full row JSON) per line.
`sortBy`	`string`	No	`none` (default), `simulation_time_seconds`, or `packet_timestamp_unix_ms` for stable ordering after merging parents.
`dataSourceName`	`string`	No	Output data source; defaults to the runner `default_store` when omitted.
`enableUploadDeduplication`	`boolean`	No	Logs output md5; skips re-upload when md5 matches prior output if enabled. Default `false`.
`lineDelimitedJson`	`boolean`	No	When `true` with `outputFormat: json`, emit one JSON record object per line (NDJSON) instead of a `records` wrapper. Required when `incrementalHydration` is `true`. Default `false`.
`incrementalHydration`	`boolean`	No	Opt-in. Stream-append only new parent outputs to cumulative NDJSON; requires `lineDelimitedJson`. Supported for `EntityDataEvent` and `TrackingEvent`. Default `false` (full replay).
`includeComponentTypes`	`string[]`	No	Allow-list of `Component` oneof arm names (for example `world_position`, `dynamics`). Mutually exclusive with `excludeComponentTypes`. Applies to `EntityState` and `SimulationFrame`.
`excludeComponentTypes`	`string[]`	No	Deny-list of component oneof arms. Mutually exclusive with `includeComponentTypes`.
`dropEmptyEntities`	`boolean`	No	When component filters are set, omit entities with zero remaining components. Default `true`.
`filterOriginUuid`	`string`	No	Exact match on capture `origin_uuid`.
`filterEntityId`	`integer`	No	Exact match on entity id (EntityState `entity.id` or event `entity_id`).
`filterEntityName`	`string`	No	Exact case-sensitive match on EntityState `entity.name`.
`entityNameRegex`	`string`	No	Python regex fullmatch on EntityState `entity.name`. AND-combined with other identity filters.
`includeDataEventTypes`	`string[]`	No	Allow-list of `EntityDataEvent` `event_data` oneof arms (for example `button_press`).
`includeTrackingEventTypes`	`string[]`	No	Allow-list of `TrackingEventType` enum names (for example `TRACKING_EVENT_TYPE_FIRED`). Applies to `TrackingEvent` and to `EntityDataEvent.event_type` when present.

When no projection filters are set, behavior matches the previous full-tree extraction.

Incremental hydration (opt-in)

Same contract as hydrate-protobuf: cumulative NDJSON per run, watermark on output config, downstream inputMode: latest. Supported for EntityDataEvent and TrackingEvent extraction.

Output shape (JSON)

The artefact is a JSON object:


{
  "extractionTarget": "EntityState",
  "records": [
    {
      "parent_input_key": "upstream-node-name",
      "payload_index": 0,
      "inner_packet_index": 2,
      "bundle_timestamp_unix_ms": null,
      "packet_timestamp_unix_ms": 1710000000000,
      "packet_timestamp_iso_utc": "2024-03-09T12:00:00.000Z",
      "origin_uuid": "…",
      "simulation_tick": 42,
      "simulation_time_seconds": 1.5,
      "capture_time_timestamp_ms": null,
      "capture_time_iso_utc": null,
      "capture_time_timezone_offset_minutes": 0,
      "extraction_target": "EntityState",
      "entity_state_index": 0,
      "entity_state": { }
    }
  ]
}

For EntityDataEvent / TrackingEvent, each record follows the decoder event dictionary layout (event_category, event_type, event_data, …) plus the same envelope fields merged in.

Example configurations

Full EntityState (legacy behavior):


{
  "extractionTarget": "EntityState",
  "outputFormat": "json",
  "sortBy": "simulation_time_seconds",
  "dataSourceName": "my-store"
}

Domain tree (world position only, named entities):


{
  "extractionTarget": "EntityState",
  "outputFormat": "json",
  "includeComponentTypes": ["world_position", "dynamics"],
  "entityNameRegex": "^(Blue|Red)-.*",
  "dropEmptyEntities": true
}

Tracking timeline for one entity:


{
  "extractionTarget": "TrackingEvent",
  "outputFormat": "json",
  "lineDelimitedJson": true,
  "incrementalHydration": true,
  "filterEntityId": 42,
  "includeTrackingEventTypes": ["TRACKING_EVENT_TYPE_FIRED", "TRACKING_EVENT_TYPE_HIT"]
}

Inputs and outputs

Inputs: All parents are merged. Each parent contributes a list of binary DataPacket payloads (same wire format as v1).
Output: One JSON or CSV artefact. Unlike v1, every parent key in fetch_all_inputs is processed, not only the first.

Core Utils Runner
Hydrate Protobuf (v1)
system-protobuf-extract (platform subset with the same projection knobs)
Node Templates

hydrate-protobuf-v2 (Transform Node)