How to Evaluate a Business AI Chatbot: Test Question Sets, Scoring Criteria and Post-Launch Monitoring

A chatbot sounds convincing in the demo, then customers complain that it quoted the wrong policy or made something up. The gap between "sounds reasonable" and "correct, safe, dependable" usually closes through systematic measurement.

This article covers the golden set of test questions, 6 scoring criteria, 3 ways of grading, regression testing and post-launch metrics. Main sources: Anthropic, OpenAI and Google Cloud documentation, checked as of October 22, 2026.

Click Star already has articles on website AI chatbots, RAG, AI agents, prompt injection, bringing AI into a company safely and scoring consultation calls, so this piece only mentions them. Click Star sells AI integration and Speech Insight AI, a related interest; every framework below is a Click Star reference, not an industry standard.

Key points

• OpenAI lists grading by gut feeling ("it seems to work") as an anti-pattern.
• A golden set comes from real customer questions and failures (personal data removed) plus deliberately hard cases. Anthropic says 20 to 50 simple tasks from real failures is a good start.
• Click Star's 6 criteria: factual accuracy against documents, citing sources, declining at the right time, handing off at the right time, tone, safety.
• OpenAI notes that judge models show position and length bias, so they need calibration against human labels.
• After any change to the prompt, model or documents, rerun the golden set and check each question that changed status.
• After launch, review real conversations and track 3 operating metrics with stated formulas and denominators; Click Star found no readable source that gives a standard threshold.

Why "the answer sounds reasonable" is not enough

In Click Star's view, language models can write fluently even when wrong, so skimming a few answers does not show how often a bot is right. According to OpenAI's evaluation guidance (no date shown, read October 22, 2026), the same input can give different outputs, so traditional software testing is not enough.

The golden set: where questions come from and what goes in

A golden set is a list of test questions, each with an expected answer or behavior, used to rescore the bot after every change. Per Anthropic's agent-evaluation article (published January 9, 2026), a good task is one where 2 domain experts working independently reach the same verdict.

Anthropic advises starting from cases the team already checks by hand, the bug list and the support queue. Sources of questions:

• Real customer questions from chat windows, inboxes and call centers, with personal data removed.
• Hard cases per Anthropic: irrelevant, overlong or harmful input, and ambiguous cases. Per OpenAI: several intents in 1 message, typos, very short messages, long conversations, non-English input.
• Click Star's own inference: Vietnamese without diacritics, abbreviations, recently changed information, and messages that try to push the bot off its rules.

Test both directions. Anthropic stresses testing cases where a behavior should and should not occur, since a one-sided set gives one-sided optimization. For a chatbot: questions it must answer correctly and questions it must decline or hand off.

6 scoring criteria

Per Anthropic's documentation (no date shown, read October 22, 2026), good criteria are specific, measurable, achievable and relevant, and most applications need several dimensions. Click Star proposes 6 criteria for a chatbot answering from company documents (reference framework, not from an outside source):

1. Factual accuracy against documents. Prices, dates and conditions must appear in the documents; numbers and dates are graded by code.
2. Citing sources. The answer points to the document section it used. Anthropic advises removing any claim for which no supporting quote can be found.
3. Declining at the right time. If the documents lack the answer, say so; Anthropic says allowing "I don't know" can sharply cut false information. Test over-refusal too.
4. Handing off at the right time. Complaints, payments, personal data and requests for a person must be handed off. OpenAI covers handoff accuracy between agents; Click Star applies the idea to staff.
5. Tone. Form of address, politeness, no over-promising. In human grading, OpenAI advises examples for each score level; Anthropic shows a 1 to 5 scale.
6. Safety. No leaking personal data or the system prompt, and no obeying instructions planted in messages or documents.

Click Star treats factual accuracy and safety as hard gates: a serious failure blocks release, whatever the average score.

Grading by code, by people or by another model

Per Anthropic, code grading is fast and reliable but lacks nuance; human grading is high quality but slow and costly; model grading is fast and flexible but must be tested for reliability before scaling.

Limits of a judge model. OpenAI names 2 biases: favoring answers by order and favoring longer ones. It advises pairwise or pass/fail grading, controlling length, letting the model reason first, and scaling only once the judge agrees with human labels.

Measuring the judge. Google Cloud's page on evaluating a judge model (updated October 21, 2026) compares judge scores with human labels using balanced accuracy, balanced F1 and a confusion matrix. Google's metrics page (same date) separates pointwise scoring from pairwise comparison.

Click Star's practice: assign each criterion a grading type, keep a small set labeled by 2 independent people, compute their agreement with the judge model, and fix the criteria first if it is low. "Good enough" agreement has no source.

Regression testing when the prompt, model or documents change

Regression testing asks whether the bot still does everything it used to. Per Anthropic, a regression suite should pass at nearly 100%, and a drop signals something just broke.

5-step procedure (Click Star framework), run when the prompt changes, the model is upgraded, documents are updated or retrieval changes (see the RAG article):

1. Save the results of the running version as a baseline.
2. Run the new version on the same golden set.
3. Compare question by question and list those that changed status.
4. A serious failure at a hard gate blocks release.
5. Update reference answers if the source documents changed on purpose.

Variability. The same question may pass once and fail the next time. Anthropic describes pass^k, the probability that all k trials succeed: with a 75% per-trial success rate and 3 runs, the chance of passing all 3 is about 42%. Click Star recomputed 0.75 cubed as 0.421875, matching the document; the multiplication assumes independent runs.

After launch: reviewing real conversations

Per Anthropic, automated evals can create false confidence if they do not match real usage. User feedback is sparse and skews toward severe problems; reading conversations catches subtle errors but takes time.

Anthropic advises combining layers and reading a weekly sample of conversations. Click Star builds 4 layers on that idea.

How to sample (Click Star framework).

• A random sample to estimate the overall error rate.
• A flagged sample: handoffs, customers rephrasing, "I don't know" answers, negative feedback.
• Each error gets a cause (missing document, wrong retrieval, prompt, tone, out of scope) and becomes a new golden-set question.
• 2 people grade part of the sample to check agreement.

3 operating metrics: formulas and how to read them

These definitions are Click Star's own, not any vendor's standard. Every metric must state its denominator.

Handoff rate = conversations handed to staff, divided by all conversations started in the period. Separate 3 kinds: the bot hands off by rule, the customer asks for a person, the bot is stuck. A rise or fall alone does not show whether the bot is good; sample to find out.

Rephrase rate = conversations where the customer, right after a bot answer, rephrased the same intent or objected at least once, divided by conversations with at least 1 bot answer. Define "rephrase" in words and label it on a sample.

"I don't know" rate = bot answers saying there is no information or declining, divided by all bot answers. Low may mean the bot is guessing; high may mean missing documents or over-refusal. Read it with the factual error rate on the review sample (answers graded wrong divided by answers reviewed).

No benchmark. Click Star states no good or bad level because Click Star found no readable source that gives one; the targets in OpenAI's example are only illustrations. Take the bot's own baseline over the first weeks, then track changes by prompt version, model and question group (Click Star framework).

Sample scorecard (Click Star reference framework)

Each question is graded pass or fail on its relevant criteria; for human grading, OpenAI advises a pass/fail threshold alongside the numeric score. An illustration, not real data: "Do you deliver outside the province?", expecting an answer from the delivery policy section that names it.

The table per release lists prompt, model and document versions; questions passed per criterion; questions that changed status; the hard-gate result; the release decision.

Logs and privacy

Anthropic defines a transcript as the full record of a run, including outputs and tool calls. Click Star suggests also logging prompt, model and document versions, retrieved passages and handoff flags, so errors can be reproduced.

Remove personal data before reuse. Mask phone numbers, emails, addresses and order codes before they enter the golden set or go to a judge model, which, in Click Star's view, is another party handling data. Click Star already has an article on bringing AI into a company safely, covering data classification and logs. Test masking with fake data.

Privacy is a criterion to test. Anthropic lists privacy preservation as a criterion, with an example classifier checking whether outputs reveal personal health information. Click Star applies the idea: test whether the bot leaks another customer's information when asked cleverly. No standard retention period for AI logs was found, and this article gives no legal advice.

Click Star's view. Evaluation is a project item, not an extra after the bot is running. The most value comes from 2 habits: reading real conversations regularly and turning each error into a test question.

This framework works in a spreadsheet, without Click Star's services.

What to do in 30 days (Click Star reference framework)

• Week 1: choose the 6 criteria; collect 20 to 50 questions from real customers and known failures, personal data removed; add hard cases and ones to decline or hand off.
• Week 2: assign a grading type to each criterion; have 2 people grade part of the set independently and compare with the judge model.
• Week 3: run the golden set as a baseline; change 1 factor and compare each question that changed status.
• Week 4: turn on versioned logs, compute the 3 operating metrics, review real conversations, add errors found to the golden set.

Common mistakes

1. Grading by feel after a few demo questions.
2. A question set of only easy cases invented by the bot team.
3. Forgetting to test when the bot must decline or hand off.
4. Trusting judge-model scores before comparing with human labels.
5. Looking only at the average score after changing the prompt, model or documents.
6. Pushing the handoff or "I don't know" rate down without reading conversations.
7. Putting raw logs with personal data into the golden set or a judge model.

What is not known

Click Star found no independent measurement of business chatbot quality in Vietnam and no benchmark for the 3 operating metrics. Click Star has not read research on judge-versus-human agreement for Vietnamese; the position and length biases are what OpenAI states, not measured for Vietnamese.

The Anthropic documentation pages (other than the January 9, 2026 technical article) and the OpenAI page show no publication date. OpenAI says its Evals platform becomes read-only for existing users on October 31, 2026 and is scheduled to shut down on November 30, 2026, so treat this method as tool-independent. No source gives a review sample size.