Skip to Content
This documentation is provided with the HEAT environment and is relevant for this HEAT instance only.
InternalRunnershydrate-protobuf incremental watermark loop

hydrate-protobuf incremental watermark loop

Summary

hydrate-protobuf incremental mode stores its parent progress watermark on the node output configuration (incrementalLastParentOutputId).

Before the fix, the incremental path could enter a loop when:

  • incrementalHydration: true
  • enableUploadDeduplication: true
  • the newly built artefact had the same md5 as the node’s current output

In that case the processor logged md5 unchanged, skipped the upload, and completed without writing a new node output. Because the watermark only advances when a new output is registered, the next run saw the same parent outputs again and repeated the same work forever.

Symptom

  • A hydrate node repeatedly processes the same parent outputs.
  • Every emitted artefact has the same md5.
  • Runner slots stay occupied and downstream nodes remain pending.

Mitigation

  • For session templates that should not use incremental replay, set incrementalHydration: false.
  • If testing a live environment before the runner fix is deployed, disabling enableUploadDeduplication also avoids the skip-upload branch.

Fix

The incremental path now continues through the upload flow even when md5 is unchanged, so the new watermark is persisted on the node output configuration and the node stops reprocessing the same parent output window.