hydrate-protobuf incremental watermark loop
Summary
hydrate-protobuf incremental mode stores its parent progress watermark on the node
output configuration (incrementalLastParentOutputId).
Before the fix, the incremental path could enter a loop when:
incrementalHydration: trueenableUploadDeduplication: true- the newly built artefact had the same
md5as the node’s current output
In that case the processor logged md5 unchanged, skipped the upload, and completed
without writing a new node output. Because the watermark only advances when a new
output is registered, the next run saw the same parent outputs again and repeated the
same work forever.
Symptom
- A hydrate node repeatedly processes the same parent outputs.
- Every emitted artefact has the same
md5. - Runner slots stay occupied and downstream nodes remain pending.
Mitigation
- For session templates that should not use incremental replay, set
incrementalHydration: false. - If testing a live environment before the runner fix is deployed, disabling
enableUploadDeduplicationalso avoids the skip-upload branch.
Fix
The incremental path now continues through the upload flow even when md5 is unchanged,
so the new watermark is persisted on the node output configuration and the node stops
reprocessing the same parent output window.