Blog Demo Join Waitlist
Back to blog

Why In-App AI Agents Need Guardrails

agents safety product

TL;DR: Giving an AI agent access to write endpoints without guardrails is like giving an intern root access on day one. Allowlists, confirmation flows, user-scoped auth, and audit logs aren't nice-to-haves. They're the reason your customers will actually turn the feature on.

Ship an in-app agent in a day

Get early access to Agent Koan and give your users a copilot that actually does things.

Join the Waitlist

The gap between "chat" and "do"

Every SaaS app is adding a chatbot. Most of them answer questions. A few can search your data. Almost none of them can do anything.

There's a good reason for that: letting an AI execute actions on behalf of a user is scary.

The models are actually quite good at understanding intent. The problem is blast radius. A wrong API call can create duplicate records, send an email to the wrong person, or delete something that shouldn't be deleted. Unlike a human clicking the wrong button, an agent can make mistakes at scale, confidently, without hesitation.

Why "just add tool calling" isn't enough

If you've used function calling or tool use in any LLM API, you know the basic pattern: describe your tools, let the model pick one, execute it, return the result. Works great in demos.

Production is not a demo. In production you need to answer:

The guardrail stack

At Agent Koan, we think about safety as a stack:

  1. Allowlists. Only endpoints you explicitly enable can be called. Everything else is off-limits. Read-only by default.
  2. Risk levels. Each tool gets a risk level (low, medium, high). Low-risk tools execute right away. Medium and high require confirmation.
  3. Confirmation UX. The widget shows a preview card with every field the agent is about to submit. The user confirms or rejects.
  4. User-scoped auth. Every API call uses the end user's token. The agent can only do what the user can do.
  5. Audit logs. Full request/response logging with latency metrics. Replay any tool call from the dashboard.

None of these are new ideas. They're borrowed from API gateways, RBAC systems, and SOC2 audit requirements. We just package them into a drop-in SDK so you don't have to build them yourself.

The point

When you have guardrails you trust, you can actually ship the feature. Your security team signs off. Your customers opt in. Your users get a copilot that doesn't just talk, it acts.

That's what Agent Koan is for. Import your schema. Configure your guardrails. Embed the widget. Ship it in a day.

Join the waitlist →

Ship an in-app agent in a day

Get early access to Agent Koan and give your users a copilot that actually does things.

Join the Waitlist