An engineering blueprint for durable agents

The demo version of an agent is usually simple. It has a loop, a model, a system prompt, an API key, and a few tools. Point it at a task and it can read scanner output, check a package, look up a CVE, and update a ticket. For vulnerability triage, that first version can be useful quickly. The work is repetitive, the inputs are uneven, and the model can connect details a human would rather not chase for the hundredth time.

Production changes the contract. The agent runs unattended on customer data, calls live integrations, handles thousands of findings, and may run in many copies at once.

Every shortcut in the demo turns into operational risk. The process that holds state can die. Credentials in the environment can leak. The model provider can return errors for an hour. The loop can follow one CVE into assets, threat intelligence, historical tickets, package paths, owners, and remediation context until it spends far more than the finding was worth. The useful output can disappear into stdout, and the container that printed it may be gone by the time anyone asks what happened.

Models are improving quickly, but a stronger model does not make an agent durable. Durability comes from the system around it. The platform has to manage state, security, retries, limits, history, and completion checks so the agent can run without a human watching every step.

What is a durable agent?

The word “agent” covers two different systems.

A personal-assistant agent sits next to a user. It runs with that user’s access, can fall back on that user’s judgment, and can ask for help because the user is present. That is useful, but it’s a different contract from unattended production work.

For vulnerability triage, “done” does not mean “the model stopped.” Done means the required work product exists. A finding has been marked false positive with evidence, or a ticket has been updated with a disposition, supporting context, and threat actor attribution. The system can check that output. If the process exits without it, the session failed.

A durable agent needs all of these.

• Its own scoped, revocable identity.
• A trigger with provenance.
• Autonomous background execution.
• Continuity across sessions.
• Human approval by exception.
• Durable work capture.
• A checkable completion condition.

The rest of the architecture exists to keep those properties true under failure, concurrency, and hostile input.

Use ephemeral workers with durable sessions

Agent workflows are highly elastic. A broad scan import can create a burst of triage work, then the fleet can fall back toward idle when the queue drains. Containers are a good primitive for this. They start quickly, isolate each execution, scale horizontally, and disappear cleanly when the work is done.

The durability problem is that the container is temporary by design. A worker can run out of memory, disappear during a deploy, lose its node, or get reclaimed while a session is halfway through enrichment. That should not lose the work, and it should not let two workers produce conflicting dispositions for the same finding.

We want to separate execution from state. The worker is disposable compute. The session is the durable record of the work. It holds status, event history, retry state, token usage, captured outputs, approval state, and completion state. Workers claim sessions, execute the next piece of work, and write transitions and artifacts back to durable storage. If a worker dies, another worker can claim the session and continue from the persisted state.

That primitive carries most of the correctness burden. A restarted controller, duplicate event delivery, recovery sweep, and two workers racing for the same job all funnel through the same conditional writes. The database chooses the winner. The other workers see that the session moved and back away.

This keeps the useful part of ephemeral workloads without making the work itself ephemeral. The fleet can scale out for a burst of findings, shrink after the queue drains, and run each worker in a clean environment. The platform preserves only the state it explicitly captures. Everything else disappears with the container.

Distinguish failure conditions

Agentic workflows will experience infrastructure turbulence.

Suppose your inference provider has transient failures for thirty minutes during vulnerability enrichment. The agent may not have received a response from the model at all. If the platform treats every failed call as an agent retry, the session can exhaust its budget before the agent has had a real chance to complete the work.

The same problem appears when a worker dies, a controller restarts, or a connection drops. Those are execution failures. They are not the same thing as failed work.

A durable system separates those failure conditions. Provider outages, process death, dropped connections, and controller restarts should re-drive the session without spending the agent’s task retry budget. The budget is spent when the agent actually attempts the workflow and fails to satisfy the task contract.

For vulnerability triage, that distinction is concrete. If the model provider returns transient errors while the agent is trying to enrich a CVE, the right behavior is to wait, re-drive, and preserve the session. If the agent completes its analysis but fails to produce a required proof artifact, finding disposition, or ticket update, that is a task failure. It should count.

The accounting matters as much as the retry mechanism. The system should know what failed. Retries should describe the task attempt instead of hiding platform noise.

Treat the agent executor as compromised

A common shortcut creates the security problem: providing the agent a vendor API key.

That key then sits inside an unattended runtime that processes user-defined inputs, invokes SDKs, runs command-line tools, and follows model-generated plans. A prompt injection can ask the agent to inspect environment variables. An ambitious agent can dump .env files or search for credentials because it’s blocked and trying to complete the task.

Assume live credentials in the execution environment will eventually leak. Build from there.

Use privilege separation. The real secret should never enter the agent runtime. The executor should receive only opaque, scoped, revocable capabilities.

In practice, the agent gets an opaque per-session token. It should look enough like a normal credential that existing SDKs and tools still work, but it should be useless outside the platform. It’s bound to one session, one tenant, a short lifetime, and a specific set of allowed hosts.

The real credential lives behind a separate egress boundary. When the agent calls an approved upstream, outbound traffic crosses that boundary. There, and only there, the opaque token is exchanged for the real credential. The upstream sees a valid request. The agent never sees the real secret.

Token indirection isn’t enough by itself. The component that runs agents should have no decryption capability for customer secrets. Not “configured not to decrypt.” No key, no decrypt path, no privileged side channel. Compromise the executor and the attacker gets an opaque token with narrow reach, not the underlying vendor secret.

Every upstream call should write an audit record, but that audit path must not become another place for secrets to leak. Store metadata such as session, scoped credential, host, time, and status. Do not store headers, request bodies, or response bodies in the credential audit path.

For vulnerability triage, the agent may need to call a scanner, ticketing system, threat-intel source, asset inventory, and source-code host. It should not hold long-lived credentials for any of them.

Keep real credentials somewhere the agent executor cannot reach.

Bound every axis of autonomy

The demo loop depends on the model’s eagerness to succeed. It explores aggressively. In production, that same behavior can become an outage.

Vulnerability triage is a good example because fan-out is common. One CVE leads to affected assets. Each asset leads to owners, package paths, reachability, exploit intelligence, historical tickets, related exceptions, and possible remediation work. A single broad scan import can become a slow, expensive crawl across the organization.

An agent that cannot be bounded still needs supervision.

Durable systems bound every axis.

• Time, with per-step and total execution limits.
• Cost, with token and spend budgets that warn softly and abort hard.
• Reach, with per-session egress allowlists and rate limits.
• Concurrency, with caps per agent, tenant, and fleet.
• Queue depth, so event fan-out cannot drown the system.
• Storage, with artifact size ceilings and retention controls.
• Process lifetime, by killing the entire process group when a step times out.

At Ghost, vulnerability workflows are bounded around exactly this fan-out problem. A broad scan result should not consume the fleet, flood upstream systems, or spend an unlimited token budget because one finding looked interesting. The system must preserve the useful autonomy while refusing open-ended exploration.

Bounds are part of durability. They keep one session from turning into a fleet problem.

Make the session replayable

If the only record is console output, the agent isn’t production grade.

This type of failure usually appears after the fact. A finding was marked false positive. A ticket was updated. A customer, auditor, or engineer asks why. The container is gone. The terminal scrollback is gone. The logs are partial, unordered, or spread across replicas. Nobody can reconstruct the work.

A durable agent leaves a replayable record. The platform persists every meaningful event, including state transitions, tool calls, tool results, token usage, approval events, artifact captures, and completion checks. Events should be ordered by server-assigned monotonic sequence numbers and not rely on the system clock of the worker.

For a vulnerability triage workflow, the replayable record should answer practical questions.

• Why was this finding marked false positive?
• What evidence supported the disposition?
• Which enrichment sources were consulted?
• What ticket update was written?
• Which scoped authority did the session use?
• Did a human approve the next step?

The point is accountability. If an autonomous system acts in your environment, you need to be able to reconstruct its work after the machine that performed it no longer exists.

Give the agent identity and memory

Every invocation starting from a blank slate is wasteful. Uncontrolled memory is dangerous. Durable agents need continuity, but continuity must be governed like any other stateful system.

Identity comes first. A durable agent should act as a provisioned, revocable, auditable principal. It should not borrow the operator’s identity. It should not self-assert who it is. Its tenant boundary should be enforced on every request and denormalized into every record so the audit trail can answer which actor acted, for whom, and under what authority.

Privilege should be narrow in two places. The agent needs a limited role inside the platform, and it needs scoped credentials for the external systems it can use. A vulnerability triage agent may need scanner read access, ticket write access, and threat-intel lookup access. It does not need broad cloud-admin privileges because a future remediation workflow might.

Memory should be just as explicit. Avoid the vague promise that “the agent remembers.” Name what persists, where it lives, how large it can grow, how it’s verified, and when it’s re-materialized into a future session.

For vulnerability triage, useful memory is local and boring. Known CVE IDs and previous dispositions can live in SQLite as session memory. Future sessions can avoid repeating analysis already completed, without depending on an external system as the only source of continuity. The agent can know that a particular CVE on a particular package path was already triaged as false positive with proof, or that a previous ticket received a specific disposition and threat actor attribution.

That is memory as operational state rather than mysticism. It is bounded, inspectable, and tied to the workflow.

Working context also belongs in durable state rather than process memory. If a session spans enrichment, disposition, approval, and remediation planning, each phase should be able to resume after worker replacement. The agent should have a past, but the platform decides what that past contains.

Make orchestration part of the state machine

Scheduling, event triggers, approvals, and completion checks often sit around the agent as orchestration code. In a durable system, that orchestration has to be part of the same persisted state machine as the session itself.

A cron entry in one container can fire twice when there are two replicas, or not at all when that container is down. A webhook handler can create thousands of sessions with no cap. An interactive approval prompt can hold a worker open for days. A clean process exit can mark a session complete even though the required output was never written.

Those are orchestration failures, but they corrupt the same product state as agent failures. The fix is the same rule used for session ownership. Each orchestration step should be a durable state transition.

Schedules should be claimed through atomic transitions. If many controllers observe that a schedule is due, only one should successfully move it to the next state. Event triggers should verify provenance before fan-out, then apply hard caps so one inbound event cannot drown the queue.

Approvals belong in persisted state as well. For vulnerability triage, a human approval might gate whether another agent opens a remediation PR. That next step can spend real time and tokens, touch source code, and create review burden for engineers. The right pattern is to pause the session, record the approval request, release the worker, and resume only when the approval arrives. If approval expires, the session moves forward as failed, skipped, or awaiting new input according to policy.

No process should sit idle waiting for a human.

Completion is also an orchestration decision. A session is complete only when the agent explicitly signals completion and the required outputs exist. For the triage workflow, that might mean a false-positive finding includes proof, or a ticket has been updated with disposition, evidence, and threat actor attribution. If the agent exits cleanly without those outputs, the process stopped. The work is not done.

Teams need this distinction before they can trust unattended work at scale. “Exited zero” is a process fact. “Completed” is a product fact.

Build the platform, not just the loop

Model capability will keep changing outside your control. Prompts and skills matter, but they are not the durable substrate. The expensive engineering lives in the system around the model. That means session state, atomic transitions, provider-aware retries, secret isolation, bounded execution, replayable history, governed memory, durable approvals, and enforced completion.

That does not mean every team should buy rather than build. Some teams should build this themselves. If agent infrastructure is core to your platform mandate, owning this substrate may be the right decision.

Most teams should be more selective. The practical question isn’t whether an agent can complete one vulnerability triage task while someone watches. The question is whether the substrate covers the production properties that matter.

• Can the session outlive the worker?
• Are retries charged to the right failure category?
• Can the agent act without receiving real credentials or secrets?
• Are time, cost, reach, concurrency, queue depth, and storage bounded?
• Can you replay what happened after the container is gone?
• Does the agent have scoped identity and governed continuity?
• Are triggers, approvals, and completion durable state transitions?

At Ghost, we invested in this infrastructure because unattended security work demands it. Vulnerability triage has too much volume to keep fully manual, but it is too sensitive to run on hope. The substrate has to carry the weight.

Three principles generalize beyond any one implementation.

• Bound every axis. An agent that cannot be bounded still needs supervision.

• Assume the agent executor is hostile. Give it opaque, scoped, revocable capabilities and keep real secrets across a boundary it cannot cross.

• Every agent state transition is conditional (e.g. compare-and-set). Make correctness under concurrency the default.

Anyone can build an agent that works once in isolation. A durable agent can keep working and produce real value even if the worker, provider, network, or human operator stops cooperating.