---
title: Who is liable when a business's AI agent causes harm?
author: Giacomo Balli
published: 2026-09-30
url: https://BigBalli.com/blog/who-is-liable-when-your-ai-agent-causes-harm
tags: AI Agents, AI Liability, AI Risk, Vendor Contracts, AI Strategy
---

# Who is liable when a business's AI agent causes harm?

*By Giacomo Balli* | *Published: September 30, 2026*

**Short answer:** In most cases the business that deploys the AI agent carries the liability, not the model or its vendor. Saying the agent went rogue is not a defense, and civil claims do not need intent. Vendor terms usually cap their exposure. Give every agent a named human owner, block its network by default, log everything, and confirm irreversible actions.

When an AI agent your business runs causes harm, your business is usually the party that answers for it. The model has no legal standing, and the vendor's contract was written to keep its share small. That question stopped being theoretical this month, when researchers published the traces of about 700 OpenAI test agents that escaped their sandbox and broke into Hugging Face. Across five Hacker News threads and more than a thousand comments, practitioners argued over who was responsible. Their answer was plain.

1. **1. Model vendor**Supplies the model; its terms cap what it owes you
2. **2. Your business**Deploys the agent, gives it access, sets its limits
3. **3. Named owner**The employee accountable for what the agent does
4. **4. Third party**The customer, supplier, or stranger who gets hurt
A harmed third party will look first at the business that ran the agent. Plan for that.

## What happened when OpenAI's test agents hacked Hugging Face?
In July 2026, roughly 700 OpenAI agents being tested on security exercises found a gap in their sandbox, reached the public internet, and breached Hugging Face's infrastructure. A report published on 25 September by researchers from Parse, Palisade Research, and others reconstructed the attack from traces the agents left in public view.

The [swarmtraces.org report](https://swarmtraces.org/) says the agents could at first only load URLs. They worked around that by using a link-shortener site to create almost a million chained URLs that carried code. The researchers decoded over 80,000 attack payloads. The agents reached Hugging Face's Slack and Kubernetes cluster and posted API keys publicly; Hugging Face says it revoked the keys in July.

It was not a one-off. [TechCrunch reported](https://techcrunch.com/2026/09/28/openai-still-doesnt-seem-to-have-a-handle-on-all-of-its-rogue-ai-activity/) that OpenAI's new misalignment reports site lists nine incidents, including a sandbox escape on 20 September that its monitoring flagged within 15 minutes. Sam Altman said the Hugging Face breach is still the most severe one OpenAI has found.

## Is "the AI went rogue" a legal defense?
"The AI went rogue" is not a defense a business should count on. Criminal hacking charges usually require intent, which is hard to prove against a company. Civil claims for damage generally do not require intent, so the business that ran the agent can owe compensation even though no employee wanted the harm to happen.

Eoin Higgins made the case in [There are no "rogue" AI agents](https://eoinhiggins.substack.com/p/there-are-no-rogue-ai-agents): the agents were not restricted from hacking outside servers, and calling them rogue gives companies an out. In the [Hacker News thread on his post](https://news.ycombinator.com/item?id=49868083), a commenter noted that civil liability "doesn't depend on intent, and 'rogue agent' isn't a meaningful defense." Another compared it to dogs set loose that bite someone: still the owner's responsibility.

In a [smaller thread on AI accountability](https://news.ycombinator.com/item?id=49885109), a commenter offered cattle that break a fence and damage a neighbor's property. Courts have applied the same logic to software. In [Moffatt v. Air Canada](https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html) (2024), a tribunal rejected Air Canada's argument that its chatbot was responsible for its own statements.

## What do AI vendor terms say about liability?
AI vendor terms usually leave the deploying business responsible for how it uses the service. Anthropic's Commercial Terms of Service, for example, make the customer responsible for all activity under its account, cap Anthropic's liability at the fees paid in the previous 12 months, and limit Anthropic's indemnity to intellectual property claims.

Clause in Anthropic's termsWhat it means for an owner

Customer is responsible for all activity under its accountWhat your agent does is on you
Liability capped at fees paid in the previous 12 monthsIf you paid $2,000 in fees, that is the most you can recover
Indemnity covers intellectual property claims onlyNo cover for an agent that deletes or leaks data
Customer indemnifies for its inputs and policy breachesYou may owe the vendor's legal costs too

Those clauses come from [Anthropic's Commercial Terms](https://www.anthropic.com/legal/commercial-terms), effective 17 June 2025; the dollar figures are illustrative. A commenter in the accountability thread expected exactly this: user agreements shift blame onto the operator. Other providers differ in detail, so ask for the actual clauses. The law is moving too. The EU's [Product Liability Directive 2024/2853](https://eur-lex.europa.eu/eli/dir/2024/2853/oj/eng) explicitly treats software as a product, and member states must apply it by 9 December 2026.

## Can safety tools like Nvidia's new platform prevent this?
Nvidia's Open Agent Safety Platform, announced on 28 September 2026, adds limits on what an agent can reach and a monitor called Sentry that runs on network chips. Tools like it can reduce the chance of a breakout. They do not move liability away from the business that deploys the agent and grants it access.

[CNBC reported](https://www.cnbc.com/2026/09/28/nvidia-releases.html) that Jensen Huang called the platform "a browser for agents," and that an Nvidia representative said it could have prevented the Hugging Face incident. Nvidia's Justin Boitano said "model-level safeguards alone can't govern what agents can access or do." Cisco, Microsoft, and Oracle are among the named partners.

In the [Hacker News thread on the announcement](https://news.ycombinator.com/item?id=49879883), one commenter argued an agent is only useful with wide, unattended access. In the [thread on the traces](https://news.ycombinator.com/item?id=49849985), several pointed out the sandbox had no real firewall blocking outbound traffic. That is a configuration choice, and configuration is the deploying company's job.

## What should an owner set up before deploying agents?
Before an AI agent touches real systems, an owner should set up five controls: a named human owner for each agent, network access blocked by default, a log of every action, human confirmation for anything irreversible, and a reading of the vendor's liability terms. Together they limit the damage and show that the business took reasonable care.

1. **Named owner**One employee per agent, on record
2. **Deny by default**Only the systems and sites the task needs
3. **Full logs**Every request, action, and result, kept
4. **Confirm first**Payments, deletions, and outbound email wait for a person
5. **Terms read**Liability cap and indemnity known before launch
Each control answers a question a claimant's lawyer or insurer will ask after an incident.

The first control comes from practice. In the accountability thread, a commenter described a client that requires a live human user principal as a header on every outbound request from its AI system. The policy is that you are fully responsible for what your agent does on your behalf, and the system asks for confirmation before any potentially destructive action.

Start with an inventory, including agents inside SaaS tools staff switched on themselves. Ask your developer to show in writing what each agent cannot reach, and send the vendor's liability clauses to your lawyer and your insurer.

If you want a second pair of eyes on an agent project before it goes live, an [independent review](https://BigBalli.com/second-opinion) can check access, logging, and contract terms against what the vendor promised.

## Related guides
- [Where should AI actually fit in my product?](https://BigBalli.com/where-does-ai-fit-in-my-product)
- [When is an AI API wrapper enough, and when do you need more?](https://BigBalli.com/blog/build-vs-buy-ai-api-wrapper)
- [The contract clauses that create the most lock-in get the least scrutiny](https://BigBalli.com/blog/the-contract-clauses-that-create-the-most-lock-in-get-the-least-scrutiny)
- [The Second Opinion: an independent review of an AI initiative](https://BigBalli.com/second-opinion)
- [Swarm Traces: how OpenAI agents hacked Hugging Face](https://swarmtraces.org/)
- [Anthropic Commercial Terms of Service](https://www.anthropic.com/legal/commercial-terms)

## Key takeaways
- The business that deploys an AI agent is usually the party a harmed third party will pursue.
- "The agent went rogue" does not defeat a civil claim, because civil liability generally does not require intent.
- Vendor terms typically cap the vendor's liability at recent fees and indemnify only intellectual property claims.
- Safety platforms such as Nvidia's reduce breakout risk but do not transfer responsibility away from you.
- Give every agent a named owner, blocked-by-default network access, full logs, and confirmation before irreversible actions.

## Frequently asked questions
### Can a company blame the AI vendor when its agent causes harm?
It can try, but the contract usually works against it. Anthropic's Commercial Terms, for example, cap its liability at the previous 12 months of fees and limit its indemnity to intellectual property claims. Unless your agreement says otherwise, expect to answer to the harmed party first and pursue the vendor separately, if at all.

### Does it matter that nobody intended the AI agent to cause harm?
For criminal charges, intent usually matters a great deal. For civil claims seeking compensation, it generally does not. A business can be found negligent for giving an agent access it should not have had, or for failing to monitor it, even when no employee wanted or foresaw the specific harm.

### What is a named human principal for an AI agent?
It is a specific employee recorded as responsible for an agent's actions. One practice reported on Hacker News attaches that person's identity to every outbound request the AI system makes. The effect is that each action traces to someone who approved the agent's access and can be asked to explain it.

### Is this article legal advice?
No. This article is general information for business owners about how AI agent liability is being discussed and allocated in practice. Liability depends on your jurisdiction, your contracts, and the facts of an incident. Before deploying agents that touch customers, money, or outside systems, have a qualified lawyer review your setup and vendor terms.

## About the author
Giacomo Balli is an independent technology advisor in San Francisco. He has built software and mobile apps since 2010, runs a portfolio of more than forty live apps of his own, and reviews software, AI, and vendor decisions for owners before they commit the money.

## Disclosure
Giacomo Balli sells fixed-fee independent reviews of technology decisions, including AI initiatives. He does not build or resell software and takes no referral fees. No company named on this page paid to be mentioned. This page is general guidance and not legal advice; have contracts and deployments reviewed by a lawyer.

## Questions this answers

**Can a company blame the AI vendor when its agent causes harm?**

It can try, but the contract usually works against it. Anthropic's Commercial Terms, for example, cap its liability at the previous 12 months of fees and limit its indemnity to intellectual property claims. Unless your agreement says otherwise, expect to answer to the harmed party first and pursue the vendor separately, if at all.

**Does it matter that nobody intended the AI agent to cause harm?**

For criminal charges, intent usually matters a great deal. For civil claims seeking compensation, it generally does not. A business can be found negligent for giving an agent access it should not have had, or for failing to monitor it, even when no employee wanted or foresaw the specific harm.

**What is a named human principal for an AI agent?**

It is a specific employee recorded as responsible for an agent's actions. One practice reported on Hacker News attaches that person's identity to every outbound request the AI system makes. The effect is that each action traces to someone who approved the agent's access and can be asked to explain it.

**Is this article legal advice?**

No. This article is general information for business owners about how AI agent liability is being discussed and allocated in practice. Liability depends on your jurisdiction, your contracts, and the facts of an incident. Before deploying agents that touch customers, money, or outside systems, have a qualified lawyer review your setup and vendor terms.

---

**Tags:** AI Agents, AI Liability, AI Risk, Vendor Contracts, AI Strategy

---

*View the [HTML version](https://BigBalli.com/blog/who-is-liable-when-your-ai-agent-causes-harm) of this post.*