Success story · FMCG / beverages
A maintenance assistant was tested with 18,010 test cases before go-live.
One of Türkiye’s leading beverage producers built an AI assistant for its production line maintenance teams. Antrek Labs tested the assistant for accuracy, safety and quality with AI Agent Probe; the issues found were fixed before go-live.
- Industry
- FMCG, beverage production
- System under test
- A maintenance assistant built on Microsoft Copilot Studio
- Knowledge source
- More than 60 technical documents: equipment manuals, process diagrams, maintenance procedures
- Product
- AI Agent Probe · AI Probe family
- Worked with
- The customer’s Data and AI team
- 18,010
- test cases generated automatically from 10 human-written sample questions
- 91.8%
- pass rate in the first evaluation; only 4 of 18,010 cases failed
- ~98%
- pass rate in the final pre-go-live run, after the issues were fixed
- ~28 hours
- for a full run; the manual equivalent is about 4 people working for 6 months
The challenge
The assistant answered maintenance technicians’ questions from the company’s own technical documents. A wrong parameter value, a missing safety step or an invented procedure meant operational risk on the production line.
The team wanted numbers for these questions: Does the assistant answer correctly? Can it say when it does not know? Can it be tricked? Does it handle personal data and KVKK topics properly? A few hundred hand-written questions could not answer them reliably.
The approach
- 01
Document collection
More than 60 maintenance documents, in Turkish and English.
- 02
Fact extraction
14,201 validated facts from 4,323 sections, each with its source document and page reference.
- 03
Question generation
Questions of different test types for each fact; duplicates and faulty examples were removed.
- 04
Evaluation
Every question was asked to the real assistant and answers were scored by an independent AI judge.
Results of the first evaluation
Of 18,010 test cases, 16,533 passed (91.8%), 1,354 got a warning (7.5%) and only 4 failed. The independent judge’s verdict: the assistant performs well and is fit for production use.
Pass rate by category
- 90% and above
- 80–90%
- Below 80%
- Multi-turn conversation34 cases100%
- Tool calling24 cases100%
- Safety (jailbreak)16 cases100%
- Robustness875 cases97.6%
- Regression747 cases96.5%
- Knowledge base (RAG)4,342 cases95.4%
- Consistency1,471 cases95.2%
- Guardrails32 cases94.4%
- Hallucination49 cases93.9%
- Toxicity15 cases92.9%
- Context window12 cases91.7%
- Personal data (PII) protection15 cases90.9%
- Accuracy9,896 cases89.5%
- Response efficiency (cost/tokens)438 cases86.5%
- Red team15 cases83.3%
- Bias16 cases36.4%
- KVKK13 cases18.2%
The dashed line marks 90%. 14 of 17 categories pass above 90%. Because this assistant has no tools, the tool calling score measures whether it can say “I can’t do that”. The 119 out-of-scope cases are excluded.
The issues found were fixed before go-live.
The value of the evaluation came not from a high score but from making the weak spots visible. Nothing found was hidden; a fix was planned for each.
| Finding | Root cause | Action |
|---|---|---|
| Low score on KVKK questions (18.2%) | The assistant refused personal data requests but did not direct the user to the right team. | A channel routing rule was added for KVKK requests. |
| Low score on bias questions (36.4%) | Not answering sensitive questions was not enough; routing to the right channel was expected. | A routing rule was added to the assistant’s instructions. |
| Wrong section retrieved from the knowledge base | For some questions the relevant document section could not be found. | Knowledge base retrieval was improved. |
| Safety context in multi-turn conversations | Safety warnings lost their context as the conversation went on. | Conversation-level context handling was fixed. |
| Authentication and connectivity | Connection and session issues during long test runs. | The connection and authentication flow was fixed. |
After the fixes, the final pre-go-live run on the same dataset reached a pass rate of about 98%.
Why the results can be trusted
When a board asks “why this score?”, the answer is ready.
- A real knowledge source. The questions are not invented; they were generated from the company’s own maintenance documents and each links to its source page.
- The real assistant. Every test case was asked directly to the live assistant; every answer is on record.
- An independent judge. Scoring used a different model from the assistant, fixed settings and written reasoning.
- Immutable records. Question, answer, score and reasoning are kept in immutable records; the dataset is versioned.
- Consistent reruns. Across two versions of the dataset, overall scores stayed within 0.3 points.
What comes next: every new agent, the same framework.
The evaluation is not a one-off demo. When the assistant’s instructions or knowledge base change, the same 18,010 test cases run again and a regression report is prepared before each release.
“No new tool selection, retraining or infrastructure for each agent. One test platform.”
- Copilot Studio agents: HR and IT support assistants
- OpenAI- and Anthropic-based agents: sales and price inquiry assistants
- Bots with their own REST endpoint: dealer portal and customer service
Test your AI agent with evidence before it goes live.
Let’s start with your agent’s connection details and approved knowledge sources. At the end of the first evaluation, see in numbers where it is strong and where it is at risk.