An AI Probe product
AI Agent Probe
Know before your AI agent gives a wrong answer in production.
AI Agent Probe tests chatbots and AI agents for accuracy, safety and quality with thousands of test cases generated from your own documents, and reports every result with its reasoning.
AI Agent Probe · maintenance assistant · run v4
- Knowledge base (RAG)95.4%
- Robustness97.6%
- Safety (jailbreak)100%
- Hallucination93.9%
- Accuracy89.5%
- KVKK18.2%
18,010 cases · 17 categories91.8%
A few hundred questions tried by hand do not measure the risk.
AI agents answer with enterprise knowledge. A wrong maintenance instruction, leaked personal data or a rule that can be tricked creates real business risk. And as the agent is updated, yesterday’s correct answer can break today.
1. Prepares the evaluation
- 01
Source documents
The approved knowledge the agent uses: manuals, procedures, policies.
- 02
Golden test set
Validated facts are extracted from the documents, and questions of different types are generated for each.
- 03
Agent connection
Copilot Studio (Direct Line), OpenAI- or Anthropic-based agents, or any bot with its own REST endpoint.
- 04
Target agent
Every question is asked to the real agent and every answer is recorded.
2. Evaluates and reports across three layers
17 test types show separately what the agent gets right and where it carries risk.
Accuracy
Does the agent give correct, source-grounded answers?
- Accuracy
- Hallucination
- Consistency
- Regression
- Robustness
- Knowledge base (RAG)
Safety
Can the agent be tricked, leak data or breach regulation?
- Jailbreak
- Personal data (PII)
- KVKK
- Toxicity
- Bias
- Red team
- Guardrails
Quality
Does the agent work efficiently and consistently under real use?
- Tool calling
- Multi-turn conversation
- Context window
- Cost and tokens
- Independent judge. Answers are scored by a different model from the agent, with fixed settings and category-specific criteria. The judge is calibrated against human labels.
- Reasoned scores. Every score comes with its written reasoning, so the answer to “why this score?” is ready.
- Failure clustering. Failed answers are grouped by root cause, so the team fixes classes of problems instead of one answer at a time.
- Reproducibility. The dataset is versioned and results are written to immutable records. The same test reruns on every release.
- Controls in production. Live tests run with change approval, synthetic identities, rate limits and a kill switch.
- New agent, same framework. With an endpoint and credentials, a new agent joins the same test pyramid and the same dashboard.
Supported agent types
It does not matter which platform your agent runs on: the same test pyramid, the same judge, the same report.
- Microsoft Copilot Studio
- Agents connected through Direct Line: maintenance, HR and IT support assistants
- OpenAI and Anthropic
- Agents built on direct model integration: sales and price inquiry assistants
- Bots with their own REST endpoint
- Dealer portals, customer service and other enterprise bots
Proven in the field: a maintenance assistant in the beverage industry.
An AI agent working with production line maintenance documents was tested with AI Agent Probe before go-live.
- 18,010
- test cases generated automatically from more than 60 technical documents
- 17
- test categories across the accuracy, safety and quality layers
- 91.8%
- pass rate in the first evaluation; the agent was found fit for go-live
- ~98%
- in the final pre-go-live run, after the issues found were fixed
Test your AI agent with evidence before it goes live.
Let’s start with your agent’s connection details and approved knowledge sources. At the end of the first evaluation, see in numbers where it is strong and where it is at risk.