How do we manage the risks arising from tool call errors?

Thanks to Prof. William Baude for sharing what happened when an AI agent deleted his notes. It started a conversation every team running agents should be having. Shared here with notes on possible technical notes.

A framework to think about the actions

1. How risky is this action?

Reading a file is low risk. Deleting client records, sending an email or moving money is not.

2. Can the action be verified?

Can we see the updated records or sent mails to check whether the action has been attempted and completed? For long running actions such as order fulfillment, do we know how and when to determine that there has been an error or failure?

3. Can it be undone?

Some actions can be restored from a snapshot or version history. Some can be staged and only committed after a check. Some, like a sent email, can’t be taken back at all.

4. What does undo cost?

Snapshots cost storage, version history costs engineering, and staging costs time. Spend that only where a mistake would cost more than the delay.

Fast path and deliberate path

The approach would depend on the risk profile of actions individually and collectively. It is not rocket science and in software we have done it many times. It is better to get into the discipline before the tools and tasks proliferate.

Low-risk, undoable actions can be in ‘fast path’. Risky or permanent ones in a ‘slower’ or ‘deliberate’ path with checks and fallbacks.

Check the regulatory context of your AI agent. ismyagentlegal.com

‍

United States
447 Broadway,
2nd Floor Suite #563,
New York 10013

Listen to Carver Conversations on Moltbook

AI Regulatory OS
Regulatory Intelligence
Regulatory Sources
Regulatory Platforms
Horizon Scanning
Regulatory Monitoring
Intelligence vs Compliance

Podcasts
Knowledge Base
Resources
Glossary
Use cases
Developers
Blog

Home
Legal Infrastructure
Regulatory Risk Intelligence
Technology
About Us

hello@carveragents.ai
General enquiries
support@carveragents.ai
Product support
legal@carveragents.ai
Privacy and legal requests
github.com/carveragents