Skip to main content
Create custom evaluation suites to batch test your agent’s performance and ensure consistent, high-quality responses across different scenarios.
AI Agent evaluation details interface

​
How evaluations work

  1. Define evaluation questions - Build a set of test questions for each agent. You can either:
    • Manually create questions that represent common use cases
    • Select responses from existing agent conversations in the Threads tab to add to your evaluation set
      AI Agent evaluation details interface
  2. Run batch tests - Execute all prompts in your evaluation set against the agent to see how it responds
    Running batch tests
  3. Review results - Manually review the agent’s responses to ensure they meet your quality standards and expectations. Lightdash also runs LLM-as-judge scorers that automatically mark each result as passed or failed and show their reasoning alongside your manual review
    Reviewing evaluation results

​
Automatic scorers

Two scorers can run on each result:
  • Factuality runs only when the prompt has an expected response. It compares the agent’s answer with your expected response, as described in Writing questions and expected responses.
  • Context relevancy runs whenever the result has context to judge: a chart or dashboard the agent produced, plus its query results when the agent has data access. It checks whether that context is relevant to the question and doesn’t use your expected response. A result passes when it scores 0.7 or higher out of 1.
A result passes only if every scorer that ran passed. You can’t turn off an individual scorer: leaving the expected response blank skips Factuality, but Context relevancy always runs when there’s chart, dashboard, or query context. If neither scorer runs, the result isn’t marked passed or failed.

​
Writing questions and expected responses

Each evaluation prompt has two fields:
  • Question (prompt) — the message you want to send to the agent, exactly as a user would type it. For example, "What's our total order revenue in 2024?".
  • Expected response — a short, plain-language description of what a correct answer looks like. This field is optional; leave it blank if you only want to eyeball responses manually.

​
What “expected response” is (and isn’t)

The expected response is not a word-for-word script the agent has to reproduce. Under the hood, when a run completes, Lightdash sends the question, the agent’s actual response, and your expected response to an LLM-as-judge that grades factual consistency, ignoring differences in style, grammar, and punctuation. The judge decides whether the agent’s answer is a subset, superset, exact match, contradiction, or an unimportant difference — and only the first three count as a pass. Because of that, the most effective expected responses are:
  • Short and factual. Describe the key facts, numbers, or behaviour the answer must include — not the full sentence you’d like to see.
  • Focused on content, not phrasing. Style, tone, and wording are ignored by the scorer.
  • Specific about numbers when you know them. If a metric should return 1,189.60, put that value in the expected response so the judge can check for it.
  • Descriptions of behaviour when there’s no single “right” number. For open-ended or ambiguous questions, describe what the agent should do (e.g. “asks for clarification”, “returns a bar chart broken down by payment method”).
You can write the expected response either as a single sentence or as a short bullet-style list of facts the answer must contain.

​
Examples

Concrete question / expected response pairs you can adapt: Specific metric with a known value
Question
Expected response
Chart or breakdown request
Question
Expected response
Ambiguous question — agent should ask for clarification
Question
Expected response
Question the agent should refuse or explain a limitation for
Question
Expected response
Empty-result question
Question
Expected response
Multi-fact answer (list style)
Question
Expected response
A good rule of thumb: if two different correct answers would both satisfy your expected response when read side by side, it’s specific enough. If either would fail because of wording alone, it’s too strict — trim it back to the underlying facts.

​
Using feedback to improve evaluations

Encourage your team to actively use the thumbs-up/thumbs-down feature when interacting with AI agents. This feedback helps admins in two key ways:
  • Identify improvement areas - Thumbs-down responses highlight where the agent needs work
  • Build better evaluation sets - Filter and easily add thumbs-down responses to your evaluation suite to test fixes and prevent regressions
Building better evaluation sets
This systematic testing approach helps you:
  • Verify agent performance before deploying changes
  • Ensure consistency across common queries