Passing evals doesn't mean your AI works

Most teams building AI agents start in the same place: evals.

Evals help validate expected behavior before launch. They catch obvious regressions and give teams confidence to ship. They're useful.

But they're not the end.

If you've ever talked to a customer support chatbot and felt yourself getting frustrated — just hoping to reach a human — you've experienced the gap.

Evals test hypotheticals.

Production reveals real usage.

The truth is: neither your team nor the model decides if an agent works correctly.

Real-world behavior over time does.

This is where monitoring comes in.

At a high level, afterlives monitors how AI agents behave in production using behavioral signals and patterns rather than raw content.

For example:

• How often do users repeat or rephrase the same request?
• How often do they abandon a flow midway?
• How often do they retry, override, or correct the system?
• Or whatever signals are important to you. No pre-designed data-sets needed!

These signals are aggregated over time to surface where agents stall, drift, or fail to resolve user needs.

Why now?

Because most AI tools were built for developers first.

First, most AI monitoring tools were built for developers. They optimize for what developers care about: correctness, reproducibility, logs and traces, debugging individual failures. The kinds of problems evals are designed to catch.

Second, we're hitting an inflection point. The first wave of AI agent companies are reaching product-market fit and scaling to real production deployments. Not 10-user pilots—hundreds or thousands of users actually relying on these systems to get work done.

Failures now don't look like bugs anymore,
but like product breakdowns:

• workarounds when the system doesn't behave as expected,
• repeated manual corrections or overrides,
• and gradual loss of trust over time.

Teams are slowly starting to use signals. You can already see early versions of this shift in products like Vercel, Replit, and Framer.

The gap we ran into

We wanted to use this kind of monitoring too, but quickly noticed a problem.

Most monitoring tools aren't built for high-stakes environments.

Because high-stakes environments come with real constraints: privacy, regulation, and user trust. PII redactions are not enough.

We felt this firsthand working in healthcare.

You can't simply log everything.
Breaking trust isn't acceptable.
But staying blind isn't either.

So we decided to build a solution.

We're designing monitoring that still works when data access is restricted or constrained, making it viable for healthcare, government, and other high-stakes environments.

We're currently running a small number of pilots.

If this resonates, we'd love to talk.