Skip to content

durable execution

A workflow (e.g. order checkout) is done in a durable fashion (fulfills even if servers go down)

essentially a durable message queue / job orchestrator with cached results (so you can tie multiple steps together and can replay steps upon workflow retry)

opensource framework: temporal

workflow: a function be to ran durably activity: a workflow step with external dependency node: server that runs the workflow and activities

checkpoint: last tracked successful state (note: it's not actually restarting from checkpoint, rather it restarts from scratch but uses cached activity results instead of re-running the already done activity)

features

  • at-least-once execution
    • the workflow will restart if incomplete until the entire workflow is completed
  • resuming
    • resume from checkpoint
    • this is important for
      • non-deterministic activity
        • (example: claude code workflow, call LLM activity, imagine node crashes and restarts session and new LLM responses were in the chat - bad experience)
      • async / long wait activity (human input)
  • abstraction
    • abstract away the distributed systems complexities of durability for any function

limitation

  • lacking at-most-once (exactly once) execution
    • more complex
    • restarting from checkpoint means the activity could have ran but node crashed before tracking activity finished, thus activity re-runs
    • compensate by making activities idempotent
  • it is not for Atomic / transactional
    • you can pair durable execution with SAGA or outbox to handle distributed transaction (if credit card step passes, but shipping fails ?)

retrying workflows

when: upon failure detection

  • timeout, heart beat, etc what:
  • retries from start but "skip" to last tracked step how:
  • log when an activity starts
  • log when an activity ends
  • cache the result of the activity that ran
  • so retry actually just replays

durable execution vs durable message queue

job orchestrator like a processing/handling a message from a Message Queue and running the job, would re-run the entire job from the start upon a node failure

whereas, durable executor is more granular - it would resume from last tracked successful state

so sub steps (activities) like

  • human approval or input
    • durable execution can simply replay from cache (instead of redoing or re-waiting)
  • non-deterministic call like LLM
    • durable execution can simply replay from cache (instead of retry leading to another value)