User Case Study

Building an AI agent that fixes its own bugs.

A senior developer at Keelo.ai used TruthAPI inside Claude Code to move from generic architecture advice to evidence-backed production design.

Signal	Why it matters
Real production problem	This is not a toy demo. The user is building runtime bug discovery/resolution for production agents used by Keelo customers.
Clear before/after	Before: ask Claude for architecture advice. After: Claude finds prior art, caveats, builders, patterns, and designs from evidence.
TruthAPI changed the user's understanding	He started thinking it was better search. He ended thinking it helps find the best prior solution and validate whether an idea will work.
The task is high-stakes	Runtime bugs in production agents involve reliability, customer impact, observability, remediation, and false-positive risk.
It surfaced one decisive practitioner	Sandipan Bhaumik became a concrete anchor: layered evals, behavioral eval, observability, and living golden datasets.
It included provenance discipline	TruthAPI did not overclaim. It said the Sandipan evidence traces to one talk, so treat it as practitioner guidance, not replicated research.
It found warnings, not just ideas	The AnyPoC / reward-hacking caveat is exactly the kind of thing a builder needs before designing auto-remediation.
It moved from research to architecture	The interaction naturally progressed from what exists, to who knows this, to design my system.
It validates design and planning	This is not about coding faster. It is about starting from accumulated engineering experience before committing to an architecture.

The problem

Keelo already had a detector for runtime bugs in production AI agents. The hard next step was a resolver: diagnose the issue, verify it, and eventually propose or ship a fix safely.

Instead of asking Claude for generic advice, the engineer asked Claude Code to use TruthAPI to find similar production systems, methods, builders, and technologies.

The prompt

I want to build a system that automatically discovers runtime bugs in my production AI agents used by Keelo customers.

Use TruthAPI to find similar production systems, people working on this problem, implementation patterns, evaluation methods, observability techniques, automatic bug discovery, automatic remediation, and anything that would help design this system.

Claude Code CLI Session

$ claude

> I want to build a system that auto-discovers runtime bugs in my production agents used by Keelo AI customers.
> Use TruthAPI to find related or similar use cases, methods, technologies, and builders who have done something similar.

Claude
I'll search for production systems, evaluation methods, observability patterns, automatic bug discovery, and remediation approaches.

tool: truthapi.getSummaries({
query: "auto discovers runtime bugs production agents bug resolver observability evaluation remediation",
sources: ["github", "docs", "youtube_video", "academic_paper", "blog_post", "discord_thread"],
total: 50
})

tool: truthapi.getSet({
ids: [
"use_case:production-error-detection-and-fixing",
"20260520T000000_product_released_academic_paper_3935d9",
"20260413T000000_insight_discovered_academic_paper_d98dd9",
"20260618T060006_insight_discovered_youtube_video_d0g5f4",
"gh-punkpeye-awesome-mcp-servers-pulls-9701_pr_opened"
]
})

TruthAPI
243 relevant items found.

Big picture:
Production AI bug-discovery systems converge around five ideas:
1. Structured execution traces
2. Layered evaluation
3. Automatic failure discovery
4. Closed-loop remediation
5. Continuous evaluation datasets

Relevant systems and methods:
- OpenTelemetry / OTLP traces
- Langfuse, LangSmith, Phoenix, OpenLIT, Braintrust
- Insights Generator for corpus-level trace diagnostics
- Hodoscope for unsupervised behavioral monitoring
- AgentEval for execution DAG evaluation
- Aient AI for OTel ingest -> grouped errors -> fix PRs

Claude
The most applicable production pattern is not "ask an LLM if this is a bug."
It is:

trace every execution
-> cluster failures across the corpus
-> detect anomalies as triage
-> synthesize deterministic reproduction
-> only then classify and fix

> Tell me more about Sandipan Bhaumik and what he has done that is relevant to my work.

tool: truthapi.related360({
query: "Sandipan Bhaumik",
hops: 2,
total: 50
})

TruthAPI
50 related concepts found.
9 directly attributed production insights found.

Sandipan Bhaumik is associated with Databricks' production AI deployment framework.
Most relevant ideas:
- evaluation before model selection
- observability as first-class infrastructure
- behavioral evaluation
- living golden datasets
- governance
- production tracing

Provenance caveat:
All attributed insights originate from a single conference talk.
Treat this as experienced practitioner guidance, not independently replicated research.

Claude
Most applicable idea for Keelo:
Evaluate the execution trajectory, not only the final output.
Many production bugs appear only in the sequence of tool calls.

> Has anyone used reinforcement learning to do this yet?

tool: truthapi.getSummaries({
query: "reinforcement learning runtime bug discovery production agents automated bug solving SWE-bench RLVR",
total: 50
})

Claude
For bug discovery: essentially no. Discovery needs an ungameable verifier, or the model can fabricate bugs.

For bug solving: yes, this is active.
Examples include Kimi-Dev-72B, BoostAPR, SWE-Shepherd, CODESKILL, and RLVR-style approaches.

Recommendation for Keelo:
Do not start by training an RL bug fixer.
Use off-the-shelf coding agents.
Invest in verifier design:
- deterministic reproduction
- golden regression cases
- semantic and behavioral evals
- bounded remediation loops

> Using everything TruthAPI found, design an architecture for my bug resolver.

Claude
Generated plan:
- runtime trace schema
- failure taxonomy
- nightly corpus diagnostics
- anomaly triage
- deterministic reproduction pipeline
- fix proposal loop
- rollback and containment strategy
- incremental roadmap

Start with deterministic verification before RL.

What TruthAPI found

production evaluation architectures

runtime observability patterns

automatic failure discovery techniques

behavioral evaluation methods

silent-failure research

production incident response patterns

automatic remediation systems

warnings about approaches that fail in practice

Decisive anchor

Sandipan Bhaumik

TruthAPI surfaced his production AI framework: layered evaluation, behavioral analysis, observability, governance, and living golden datasets.

Trust builder

Provenance caveat

TruthAPI also warned that the Sandipan evidence traces to a single conference talk: useful practitioner guidance, not independently validated research.

Critical warning

Verify before fixing

The AnyPoC / reward-hacking evidence changed the design: do not trust an LLM to declare bugs. Require deterministic reproduction before remediation.

"It validated whether the idea I have will actually work."

"It told me the tools I'd need, whether it has been done before, and the approaches people are actually using."

Senior developer, Keelo.ai