Passing evals doesn't mean your AI works

Most teams building AI agents start in the same place: evals.

Evals help validate expected behavior before launch. They catch obvious regressions and give teams confidence to ship.

They're useful.
But they're not the end.

If you've ever talked to a customer support chatbot and felt yourself getting frustrated — just hoping to reach a human — you've experienced the gap (now imagine the same thing in a safety-critical environment).

Evals test hypotheticals.
Production reveals real usage.

The truth is: neither your team nor the model decides if an agent works correctly.
Real-world behavior over time does.

This is where AI monitoring comes in.

At a high level, afterlives monitors how AI agents behave in production using behavioral signals and patterns.

For example, how often do users...

• repeat or rephrase the same request?
• abandon a flow midway?
• click thumbs down?
or
• how often are there high risk interactions?

These signals are aggregated over time to show whether recent changes made the system better or worse in real use, and where agents stall, drift, or fail.

Why now?

In the first wave, AI systems were mostly offline, internal tools, or narrow chatbots. Evals were enough, production risk was low.

In the second wave, AI agent companies started reaching product-market fit and scaling to real production deployments. Instead of 10-user pilots, there are hundreds or thousands of users actually relying on these systems to get work done.

Failures no longer look like bugs. They look like silent product breakdowns:

• workarounds,
• repeated corrections,
• abandoned flows,
• and gradual loss of trust over time.

That's where behavioral monitoring started to emerge. Companies like Vercel, Replit, and Framer all integrated tools into their workflows.

Now we're entering the third wave: AI agents operating in high-risk and regulated environments.

Here is where we noticed the gap.

The gap we ran into

We wanted to use this kind of monitoring too, but quickly noticed a problem.

Most AI monitoring tools assume you can log everything. But if you're building AI for healthcare, finance, or government, you can't.

Privacy regulations. Compliance requirements. Customer trust. These aren't nice-to-haves. We felt this firsthand working in healthcare.

Traditional monitoring tools force an impossible choice:

• Log everything → violate privacy, break trust, fail audits
• Log nothing → fly blind, no defensible oversight, react to incidents after they happen

PII redactions are not enough if your AI touches insdustries like healthcare, finance, or government.

Everyone's starting to monitor AI agents now—except where it matters most. Where your users are non-AI-native and one undetected failure pattern can kill a six-figure contract.

And where regulatory frameworks like the EU AI Act now require continuous post-deployment monitoring of high-risk systems.

You need production monitoring that respects the constraints you operate under. We are running select pilots now. If you're building AI for regulated industries, let's talk.