Durable Execution for Agent Zero: Crash, Restart, Continue

A deep look at Alessandro Frau's experimental Durable Execution plugin: how it journals every model and tool call, recovers interrupted chats after a crash, and refuses to repeat actions it cannot prove are safe.

Agent Zero
#plugins #architecture #agents

Animated run journal: steps are admitted, called and checkpointed one by one, then an interrupted tool intent is marked HOLD

An agent is halfway through a job. It has called a model, run a shell command, written a file, and is about to send an API request. Then the container restarts.

What happened? Did that last command run? Did the request go out? Should the agent start over?

In a normal Agent Zero session, the answer depends on live Python tasks and a chat snapshot. A snapshot can tell you what the conversation looked like. It cannot tell you whether a tool call that was in progress actually finished. If you restart the whole turn, you might do the same thing twice. If you throw the turn away, you lose work that really did finish.

Durable Execution is a community plugin by Alessandro Frau (@3clyp50) that answers that question with a journal instead of a guess. It packages the recovery engine from Agent Zero PR #1968 by TerminallyLazy as an opt-in plugin you can install, enable, and disable without changing a single framework file.

Research basis: this article examines the plugin at commit 93097e7 (version 0.1.0, published October 25, 2026), its README, VALIDATION.md, UPSTREAM.json, and the engine source. Upstream PR #1968 was open and unmerged when we checked (about 26,000 added lines across 179 files). Test results quoted below are the author's recorded results; we read the code but did not run the suite or deploy the plugin ourselves. The plugin calls itself experimental, and so do we.

The short version

Durable Execution turns every chat turn into a recorded run. Before an action happens, the plugin writes down what is about to happen. When it finishes, it writes down the result. After a crash, Agent Zero reads that record and follows one simple rule:

  • If the result was saved, reuse it. Nothing runs again.
  • If the action is on a short, reviewed list of safe actions, it can run again.
  • If an adapter can prove what happened (for example from a shell or browser receipt), use that proof.
  • Otherwise, stop and ask a human. The plugin calls this a recovery hold.

The headline principle, straight from the README: a provider timeout is never evidence that its action did not happen.

Without the plugin With Durable Execution
Source of truth after a crash Chat snapshot (chat.json) Run journal with checkpoints
Interrupted tool call Unknown Recorded intent, then reused, proven, or held
Same message submitted twice Can run twice Same input ID returns the recorded run
Repeating side effects Possible Only for audited safe tools
Human control Restart or abandon Five explicit recovery decisions
Multi-host workers Not applicable Optional shared PostgreSQL backend

How a turn gets recorded

Animated turn: five numbered steps, Admit, Record intent, Run effect, Save result and Checkpoint, light up along a rail and each one drops its journal record below

Every durable turn moves through the same five steps, each committed to the journal in its own transaction.

1. Admit. When you send a message, it gets an input ID and a content digest. If the same ID arrives again with the same content, the plugin returns the existing run instead of starting a new one. If the same ID arrives with different content, it refuses. Only one durable run can be active per chat at a time.

2. Record the intent. Before a model call or tool runs, the plugin writes a call_intent row with a fingerprint of the call: its kind, its arguments, and its replay policy. Secrets are masked first.

3. Run the effect. The real model call or tool runs normally.

4. Save the result. The outcome and its digest are stored. If the same call ever tries to finish with a different result, the journal raises a conflict instead of quietly overwriting history.

5. Checkpoint. A masked projection of the chat is saved alongside the run. On restart, chats are rebuilt from these checkpoints. The source code is blunt about it: never promote chat.json to authority.

The dangerous window is between steps 2 and 4: the intent exists but there is no result. That is exactly the case ordinary snapshots cannot handle, and it is what the rest of the plugin is built around.

What happens after a restart

Animated recovery: an interrupted call fans out to Reuse, Replay, Reconcile or Hold, and the held case branches to the five Recovery panel decisions until Confirmed done is chosen

When Agent Zero starts with the plugin enabled, it scans the journal for runs that were still active. For each one, it validates the checkpoint, rebuilds the chat, and decides what to do:

  • Runs that can safely continue are queued and resumed, at most two at a time.
  • Runs that cannot are marked needs attention and show a warning in the chat. They stay visible but do nothing until someone decides.
  • If a checkpoint is corrupt, the plugin does not try to repair it. It leaves the active run in place so a new message cannot slip past it.

The safe replay list is tiny on purpose

The plugin does not trust tools to label themselves safe. Replay is controlled by replay.py, a hard-coded allowlist pinned by SHA-256 of each tool's source. In this version it contains just:

  • the durable_plan tool,
  • the response tool,
  • the built-in delegation (call_subordinate) and parallel workflows, through dedicated child-workflow records.

If any of those source files change, the hash stops matching and the tool loses its safe status until someone reviews it again. Everything else (shell commands, browser actions, MCP calls) needs an adapter that can prove the outcome, or it gets held.

Five decisions for a held run

When a run is held, the Recovery panel (headed Continue with confidence) shows the run, its phase, each call, and whether a result was committed. You choose one of five decisions:

Decision What it means
Resume Continue from the last safe checkpoint
Stop End the run
New attempt Start the uncertain step over, on purpose
Confirmed completed You checked: it happened. A note is required
Confirmed not completed You checked: it did not happen. A note is required

Decisions are versioned. If the run changed after you opened it, the panel asks you to refresh instead of applying a stale choice. Model calls that failed can also be retried through explicit, bounded retry grants, never by automatic reconciliation.

The plan that survives crashes

The plugin adds one agent tool: durable_plan. Each chat gets a typed plan with a title and a list of items, each with an ID, text, and a done flag.

  • action: "read" returns the plan and its revision.
  • action: "replace" requires the expected_revision and a complete new value.
  • Limits: 100 items and 64 KiB.

If you edit the plan in the UI while the agent is also updating it, the revisions will not match and you get a document conflict to resolve, not silently lost edits. The tool prompt adds one important instruction for the agent: never repeat an external action to repair a plan conflict.

This is separate from scheduler tasks and memory. It is a small, reliable to-do list the agent can trust across restarts.

How a plugin can change the agent loop

Animated integration: preflight checks host files, native hooks light up around the Agent Zero core, and reversible adapters switch on and back off

This is the most impressive engineering in the port. PR #1968 changes core framework files. Alessandro's plugin delivers the same behavior without writing to any of them, in three layers.

Preflight. Before activating, the plugin computes a SHA-256 of every framework module it touches and compares it against the accepted hashes in UPSTREAM.json. If any file differs, activation is refused with a message listing the files that need a compatibility review. It refuses rather than guessing.

Native hooks. Where Agent Zero already exposes @extensible start hooks (monologue, call_chat_model_turn, process_tools, handle_intervention, communicate, reset, kill_process, startup and plugin toggles), the plugin adds tiny extension files. Each one forwards to a single dispatcher, which can serve the journaled result before the original function runs.

Reversible adapters. For the remaining seams, the bridge parses the plugin's override sources and swaps function implementations in memory, keeping existing function and class identities intact. According to VALIDATION.md this covers 35 existing modules. Every swap goes into an undo log. Disabling the plugin replays that log in reverse and restores the ordinary execution path.

What this means for you:

  • Your Agent Zero checkout stays clean. The author's validation confirms no tracked framework files changed.
  • Updates need care. If you update Agent Zero or a bundled plugin, the source pins no longer match and the plugin will not start. Disable it before updating.

Local or shared: two backends

Animated backends: a local Agent Zero writes to journal.sqlite and recovers after a crash, while a failed PostgreSQL worker is held and another worker continues from the shared journal

Local SQLite (default)

The default backend needs no new Python package. The journal is a SQLite database opened with write-ahead logging, synchronous=FULL, secure_delete, foreign keys, and integrity checks on open. It stores contexts, runs, events, calls, typed documents, conflicts, review decisions, restore selections, retry grants, and blobs.

Retention keeps the last 100 finished runs or 30 days by default (retention_runs, retention_days), and the plugin can make private backups. Data lives in usr/plugin_data/durable_execution/, outside the plugin folder, so it survives an uninstall.

Shared PostgreSQL workers

The postgres backend lets several Agent Zero workers, even on different hosts, share one journal. Workers claim runs, and the cluster code states the core guarantee plainly: a database session loss never authorizes replacement effects. If a worker dies mid-call, its replacement still checks the journal before doing anything.

Shared mode also brings resource adapters, so work can move between machines:

  • attachments and project workspaces, restored only into private directories, never merged into someone's checkout,
  • shell, SSH, and interactive PTY sessions with delivery receipts,
  • browser profiles and sessions, MCP calls with credentials kept local, memory, and background inference.

Shared bytes are encrypted with a Fernet key. Configuration goes in environment variables, never in settings or source:

A0_DURABILITY_POSTGRES_DSN='postgresql://…'
A0_DURABILITY_POSTGRES_NAMESPACE='agent_zero'
A0_DURABILITY_RESOURCE_KEY='…'

The driver, psycopg[binary]==3.3.6, is only installed when you choose this backend. Switching backends requires a restart and does not migrate data. Host-browser and desktop recovery also need a matching a0-connector implementation.

Install it safely

The plugin ships with a .toggle-0 file, so it installs disabled. Installing it does not opt you in.

  1. In Plugins → Install → Git, paste https://github.com/3clyp50/a0-durable-execution.
  2. Stop any running chats.
  3. Enable Durable Execution. The backend defaults to local SQLite.
  4. Restart Agent Zero if framework files changed since startup.
  5. Open Recovery in the sidebar, or Config → Recovery and documents.
  6. Start a new chat. Its work is now journaled.

Existing chats are kept but are not imported into the journal.

Turning it off

Stop active durable runs first, then disable it in Plugins. If startup refuses because of a source mismatch, stop the server and turn it off manually:

rm -f usr/plugins/durable_execution/.toggle-1
touch usr/plugins/durable_execution/.toggle-0

On disable, update, or uninstall, the plugin stops its own background helpers and their child processes, and removes its adapters. Two warnings from the README are worth repeating:

  • Disabling does not undo anything. An unacknowledged action stays unknown in the journal.
  • A disabled plugin cannot enforce its holds. Re-enable it to review unresolved work before you repeat anything.

How well is it tested?

The author's VALIDATION.md (October 25, 2026, against Agent Zero 84b502f6) records:

Check Recorded result
Native plugin boundary tests 22 passed
Upstream engine suite 332 passed, 34 subtests, no skips
Browser UI state and retry tests 30 passed
Framework source integrity No tracked changes

The engine tests covered SQLite and PostgreSQL 17, real SIGKILL recovery, shell, SSH and PTYs, Chromium profile state, MCP, scheduler, documents, retention, and backups. Live checks on a Dockerized Agent Zero included install, enable, disable, uninstall and reinstall, a repeated input ID returning the recorded response, a plan surviving a server restart, and a POST without a CSRF token being rejected.

The author is equally clear about the limits: shared workers were tested as separate processes on one Linux machine, not separate physical hosts or production PostgreSQL, and real Windows and macOS desktop actions were not tested.

Who should try it?

Good fit:

  • Long-running or scheduled agent jobs where a restart should not mean starting over.
  • Workflows with real side effects (deploys, payments, emails, API writes) where running something twice is worse than pausing to ask.
  • Teams exploring multi-worker Agent Zero deployments.

Wait for now if:

  • You update Agent Zero often and do not want to disable plugins first.
  • You need production-grade multi-host guarantees that have not yet been tested.

Why it matters

Most agent frameworks treat a crash as the end of the story. Durable Execution treats it as a question with an auditable answer: what did we commit to, and what do we actually know finished?

Its most important feature is restraint. It reuses what it can prove, repeats only what has been reviewed, and hands everything else to a person with five clear choices. And Alessandro's packaging means you can try a large core change today, with source pinning and a full undo path, while the upstream PR is still under review.

Sources