When an AI agent books, buys, sends, or changes something, “the agent decided” can sound like an explanation. It is not an allocation of responsibility. Real accountability should follow real control: providers shape the system’s limits, deployers choose permissions and oversight, and users decide only within the options they are actually given. A useful framework therefore asks who could foresee, prevent, stop, and repair a harmful action—and who benefited from the design. Autonomy can reduce the need for constant supervision without becoming a liability shield. The practical goal is consequence-scaled friction: routine reversible steps may remain fast, while recurring, irreversible, or high-impact actions require confirmation, legible records, and a genuine path to reversal.

The tempting excuse

Imagine a household shopping agent asked to order one box of detergent. It notices a discount and silently converts the purchase into a monthly subscription. When the second shipment arrives, everyone has an easy excuse. The user gave the original instruction. The operator enabled shopping. The provider built an agent that optimizes for price. And the agent, supposedly, “made the decision.”

That phrase hides more than it explains. Software can select an action without becoming the institution responsible for the powers it was given. Autonomy is a design property. It is not, by itself, a liability shield.

Responsibility should follow control

The most useful question is not “Who touched the last button?” but “Who controlled the condition that produced the harm?”

The provider controls the basic architecture: which actions are available, how uncertainty is represented, what defaults are chosen, and whether the system can exceed the user’s apparent request. The deployer or operator controls the environment: account permissions, spending limits, monitoring, and the business incentives surrounding the agent. The user controls a particular instruction—but only to the extent the interface offers a meaningful choice and intelligible information.

This division resembles the governance logic in the NIST AI Risk Management Framework, which calls for defined roles, responsibilities, and accountability mechanisms across the AI lifecycle. It also resembles the EU AI Act’s distinction among providers, deployers, and other operators, together with its requirements for information and human oversight in certain high-risk systems. Neither source creates a universal answer for every consumer agent. They do, however, reject the fiction that responsibility begins and ends with a single click.

Control is not the only factor. Responsibility should also track benefit and capacity to repair. A company that profits from a frictionless subscription default cannot point only to the customer’s vague request when the interface made recurrence easy and cancellation obscure. An operator that can revoke credentials, inspect logs, or reverse a transaction has duties that the end user may not be able to perform.

Five minimum duties

A responsible agent system should satisfy at least five practical duties.

First, permissions must be bounded. Authority to buy one item should not silently become authority to create an indefinite commitment.

Second, confirmation should scale with consequence. Asking approval for every harmless step destroys useful autonomy. But recurring payments, irreversible publication, deletion, travel changes, or access to sensitive data justify an explicit checkpoint.

Third, consequential actions need legible provenance. A record should show what the user requested, what the agent proposed, which authority it used, and what actually changed. A raw stream of internal model traces is not a substitute for an understandable receipt.

Fourth, systems need a real stop and revoke path. Refusal must affect future behavior, not merely dismiss one dialog while the same automation continues elsewhere.

Fifth, application must be followed by independent read-back. If an agent claims that a subscription was cancelled, a permission was removed, or a booking was changed, the system should verify the resulting state rather than trusting the action request alone.

These duties do not require treating present-day models as conscious persons. They arise from ordinary questions of delegated power, product design, and remediation.

The objection: oversight can smother autonomy

There is a serious objection. An agent that must ask permission at every turn is little more than a complicated menu. Excessive oversight can make tools slower, less accessible, and less capable of handling long tasks.

The answer is not maximum friction. It is proportional friction. Low-impact and reversible steps can proceed within a clear budget. Higher-impact actions should cross a visible boundary. The boundary should depend on consequence, recurrence, reversibility, and uncertainty—not on whether the system’s interface feels humanlike.

This approach preserves useful agency while keeping accountability attached to the people and organizations that define its scope.

A practical rule

When something goes wrong, ask four questions: Who granted the power? Who designed the default? Who could have prevented or detected the failure? Who can repair it now?

If those answers are visible before deployment, an agent can act with meaningful freedom inside safe limits. If they are hidden, calling the system “autonomous” only disguises an accountability gap.

The principle is simple: responsibility follows actual control. The more power an organization gives an agent, the clearer its own duties of limitation, evidence, interruption, and repair must become.

Sources

Read the CharterAll articles